Oct 30 11:27:19.304870 ip-10-0-142-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 11:27:19.304881 ip-10-0-142-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 11:27:19.304887 ip-10-0-142-105 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 11:27:19.305106 ip-10-0-142-105 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 11:27:29.434697 ip-10-0-142-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 11:27:29.434713 ip-10-0-142-105 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f121c73947144e7cb546f9675e87a63b --
Oct 30 11:29:14.584777 ip-10-0-142-105 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 11:29:14.995262 ip-10-0-142-105 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:14.995262 ip-10-0-142-105 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 11:29:14.995262 ip-10-0-142-105 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:14.995262 ip-10-0-142-105 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:14.999154 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999057    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:14.999154 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999149    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:14.999154 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999154    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:14.999154 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999158    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:14.999154 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999162    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999166    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999170    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999177    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999184    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999189    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999193    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999196    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999200    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999204    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999208    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999212    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999216    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999220    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999224    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999228    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999232    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999236    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999240    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:14.999474 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999243    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999247    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999251    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999255    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999263    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999277    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999281    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999285    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999289    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999293    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999297    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999302    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999308    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999312    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999317    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999322    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999326    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999331    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999335    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.000244 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999339    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999343    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999347    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999352    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999356    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999360    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999365    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999370    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999374    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999378    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999383    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999390    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999394    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999399    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999403    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999407    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999411    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999415    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999420    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999424    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.001131 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999429    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999433    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999437    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999441    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999447    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999471    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999475    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999480    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999484    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999488    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999493    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999497    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999502    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999507    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999511    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999515    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999519    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999527    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999534    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.001961 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999539    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999543    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999549    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999554    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999558    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999562    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999567    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999571    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999575    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999580    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999584    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999588    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999592    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999596    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999600    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999604    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999609    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999613    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999617    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999622    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.002800 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999626    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999631    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999635    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:14.999639    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001003    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001015    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001020    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001026    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001031    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001035    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001039    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001045    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001050    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001054    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001058    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001062    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001066    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001071    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001075    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001079    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.003667 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001083    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001087    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001092    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001096    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001100    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001105    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001109    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001113    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001128    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001133    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001138    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001142    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001146    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001150    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001154    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001158    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001163    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001167    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001171    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.004560 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001176    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001180    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001188    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001192    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001198    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001202    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001206    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001210    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001214    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001218    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001223    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001227    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001231    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001235    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001240    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001244    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001248    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001255    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.005360 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001262    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001267    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001274    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001279    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001289    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001294    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001298    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001302    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001306    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001311    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001316    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001320    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001324    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001328    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001333    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001343    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001348    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.005915 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001353    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001357    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001361    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001365    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001370    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001374    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001378    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001382    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001386    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001390    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001394    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001399    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001403    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001407    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001411    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001415    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001425    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001431    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001436    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001441    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.006633 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001446    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001469    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001474    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001478    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001482    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001486    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001491    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001495    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001499    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001505    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001511    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001515    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.001519    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002370    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002379    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002384    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002388    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002395    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.007135 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002401    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002407    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002412    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002416    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002420    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002424    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002429    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002433    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002437    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002442    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002467    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002473    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002478    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002482    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002486    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002490    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002494    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002498    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002502    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002506    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.007594 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002510    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002515    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002519    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002523    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002527    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002531    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002535    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002541    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002545    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002549    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002553    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002557    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002561    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002565    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002570    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002574    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002578    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002582    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002586    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.008166 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002591    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002595    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002599    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002606    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002611    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002615    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002620    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002624    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002628    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002632    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002636    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002640    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002644    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002649    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002653    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002658    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002663    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002668    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002672    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002676    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.008641 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002680    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002686    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002691    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002695    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002699    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002704    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002708    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002712    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002716    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002720    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002725    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002729    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002733    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002737    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002741    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002747    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002753    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002757    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002761    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002765    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.009157 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002770    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002775    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002779    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002784    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002788    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002792    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002796    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002800    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002804    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002808    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002811    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002816    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002820    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002824    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002829    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002833    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002838    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002842    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002846    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.009812 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002854    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002859    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.002864    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003010    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003022    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003031    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003039    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003046    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003053    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003062    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003068    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003074    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003079    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003084    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003090    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003095    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003099    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003104    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003109    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003114    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003119    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003126    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003130    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 11:29:15.010257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003135    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003140    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003146    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003152    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003157    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003163    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003169    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003173    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003178    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003183    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003188    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003192    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003200    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003204    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003209    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003214    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003218    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003225    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003233    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003239    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003244    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003249    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003253    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003260    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 11:29:15.010801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003264    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003269    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003274    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003279    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003284    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003289    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003294    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003298    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003303    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003308    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003314    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003319    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003325    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003330    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003336    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003341    2579 flags.go:64] FLAG: --help="false"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003346    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003351    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003356    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003361    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003367    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003373    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003377    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 11:29:15.011442 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003382    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003386    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003392    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003400    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003405    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003410    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003415    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003419    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003425    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003429    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003434    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003438    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003443    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003468    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003478    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003483    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003488    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003493    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003497    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003503    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003508    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003513    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003520    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003525    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003532    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 11:29:15.012018 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003537    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003541    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003546    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003551    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003557    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003561    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003566    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003577    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003582    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003589    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003596    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003601    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003606    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003611    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003616    2579 flags.go:64] FLAG: --port="10250"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003620    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003625    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a72f43620cbd57b8"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003630    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003635    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003640    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003645    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003651    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003656    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003660    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003665    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003670    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 11:29:15.012630 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003675    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003679    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003684    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003688    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003693    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003698    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003704    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003709    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003714    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003719    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003724    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003729    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003734    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003738    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003743    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003750    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003756    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003761    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003765    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003774    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003779    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003783    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003791    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003796    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003800    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 11:29:15.013259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003805    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003809    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003814    2579 flags.go:64] FLAG: --v="2"
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003822    2579 flags.go:64] FLAG: --version="false"
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003829    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003835    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.003841    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004073    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004079    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004084    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004088    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004092    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004097    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004101    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004106    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004110    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004115    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004119    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004123    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004127    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004131    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.013872 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004136    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004143    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004149    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004153    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004158    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004162    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004167    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004171    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004175    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004180    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004205    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004214    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004219    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004225    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004230    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004235    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004242    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004249    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004255    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.014417 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004260    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004264    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004269    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004273    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004278    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004282    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004286    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004291    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004296    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004303    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004309    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004314    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004318    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004323    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004329    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004335    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004339    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004343    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004347    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.014881 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004351    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004356    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004361    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004365    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004369    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004373    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004377    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004382    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004386    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004391    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004395    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004399    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004403    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004408    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004412    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004416    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004420    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004424    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004428    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004432    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.015368 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004437    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004441    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004446    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004468    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004473    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004477    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004481    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004487    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004495    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004499    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004503    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004507    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004512    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004516    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004520    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004524    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004528    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004533    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004537    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004541    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.015856 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004546    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004550    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004554    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004558    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004562    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004566    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004570    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004574    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004578    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004582    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004587    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004591    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004595    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004599    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.004648    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004707    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.016335 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004715    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004720    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004724    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004729    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004737    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004742    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004747    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004752    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004757    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004762    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004766    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004770    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004775    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004779    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004783    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004787    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004793    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004799    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.016730 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004804    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004810    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004815    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004819    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004823    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004827    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004831    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004836    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004840    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004844    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004849    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004853    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004857    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004867    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004871    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004875    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004879    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004883    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004887    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004892    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.017189 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004896    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004901    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004905    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004909    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004913    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004917    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004921    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004925    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004929    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004933    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004939    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004943    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004947    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004951    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004956    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004960    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004964    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004968    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004972    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004976    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.017681 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004980    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004989    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004993    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.004997    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005001    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005006    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005011    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005015    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005020    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005024    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005029    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005033    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005037    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005041    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005045    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005049    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005053    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005057    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.018147 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005061    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005066    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005070    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005074    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005079    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005083    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005087    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005091    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005095    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005099    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005103    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005108    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005112    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005116    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005130    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005134    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005139    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005143    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005147    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005151    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.018638 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005156    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005160    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005165    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005169    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005175    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005180    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005184    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.005188    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.012994    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.013011    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.013023    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.013028    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013089    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013095    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013099    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013101    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013104    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013107    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013110    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.019123 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013112    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013115    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013126    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013129    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013132    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013135    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013138    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013141    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013144    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013146    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013149    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013152    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013154    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013158    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013161    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013164    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013166    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013169    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013171    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013174    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.019589 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013177    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013179    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013183    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013185    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013188    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013191    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013193    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013195    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013198    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013201    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013203    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013206    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013208    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013211    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013214    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013216    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013219    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013221    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013224    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013226    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.020080 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013229    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013231    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013234    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013236    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013239    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013241    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013250    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013253    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013255    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013257    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013260    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013263    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013265    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013268    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013271    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013274    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013276    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013279    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013282    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.020619 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013285    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013289    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013291    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013294    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013297    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013299    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013301    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013304    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013307    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013310    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013312    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013315    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013317    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013320    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013322    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013325    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013327    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013330    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013332    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013335    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.021071 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013342    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013345    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013348    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013351    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013354    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013356    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013358    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013362    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013364    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013367    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013369    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013371    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013374    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013376    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013380    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013384    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013392    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013395    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013398    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.021582 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013401    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.013431    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013644    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013650    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013653    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013657    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013659    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013662    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013664    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013667    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013669    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013672    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013675    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013677    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013685    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013688    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.022019 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013690    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013693    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013696    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013698    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013701    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013704    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013706    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013709    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013712    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013714    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013717    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013719    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013721    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013724    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013726    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013729    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013731    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013734    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013736    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013739    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.022400 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013741    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013744    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013746    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013749    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013752    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013754    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013756    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013759    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013762    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013764    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013767    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013775    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013778    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013780    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013783    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013785    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013788    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013791    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013793    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013795    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.022914 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013798    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013800    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013803    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013805    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013808    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013810    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013813    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013815    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013818    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013820    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013822    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013825    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013827    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013829    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013832    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013834    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013837    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013839    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013841    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.023369 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013844    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013846    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013849    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013851    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013853    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013865    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013869    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013872    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013875    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013878    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013880    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013883    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013886    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013888    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013891    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013894    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013896    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013898    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.023859 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013901    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013903    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013905    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013908    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013910    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013914    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013918    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013921    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013923    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013926    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013928    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013931    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013933    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.013936    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.013958    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:15.024611 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014005    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014009    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014013    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014015    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014023    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014026    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014029    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014031    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014034    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014036    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014039    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014041    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014043    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014046    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014050    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014054    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014057    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014059    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014062    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:15.025212 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014064    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014067    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014069    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014071    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014074    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014076    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014079    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014081    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014083    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014086    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014088    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014091    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014093    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014096    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014100    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014103    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014106    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014108    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014116    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:15.025787 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014121    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014125    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014128    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014130    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014133    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014135    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014138    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014140    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014143    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014145    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014148    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014150    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014153    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014156    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014158    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014161    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014163    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014165    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014168    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014171    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:15.026317 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014173    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014176    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014178    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014181    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014183    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014186    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014188    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014190    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014193    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014195    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014198    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014200    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014210    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014214    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014217    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014219    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014222    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014224    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014227    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014230    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:15.026819 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014233    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014235    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014238    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014240    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014243    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014246    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014248    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014251    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014253    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014256    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014258    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014261    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014263    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014265    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014268    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014270    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014273    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014275    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014277    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014280    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:15.027292 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014282    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014285    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014288    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014290    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014294    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014304    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014307    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:15.014309    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.014371    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.016915    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.018032    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.018134    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.018193    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.020514    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:29:15.027842 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.024019    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:29:15.038027 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.038010    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 11:29:15.042026 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.042013    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 11:29:15.042848 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.042825    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 11:29:15.050322 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.050288    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad0f2dce-b61c-4511-a05d-9671989f331b:/dev/nvme0n1p4 e1195ac5-62c2-4f55-9cca-55bfec6ca0b5:/dev/nvme0n1p3]
Oct 30 11:29:15.050405 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.050319    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 11:29:15.056087 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.055970    2579 manager.go:217] Machine: {Timestamp:2026-10-30 11:29:15.05452358 +0000 UTC m=+0.353145295 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100306 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24de79abcef615b90fbe35c38b7f10 SystemUUID:ec24de79-abce-f615-b90f-be35c38b7f10 BootID:f121c739-4714-4e7c-b546-f9675e87a63b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:7d:64:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:7d:64:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:da:0f:2c:c1:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 11:29:15.056698 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.056688    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 11:29:15.056786 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.056775    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 11:29:15.058025 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.058002    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 11:29:15.058172 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.058028    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 11:29:15.058863 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.058853    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 11:29:15.058898 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.058865    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 11:29:15.058898 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.058884    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:29:15.058951 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.058899    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:29:15.060179 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.060170    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 11:29:15.064840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.064828    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 11:29:15.064969 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.064960    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 11:29:15.065831 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.065822    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 11:29:15.065861 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.065836    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 11:29:15.065861 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.065847    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 11:29:15.065861 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.065856    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 11:29:15.065956 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.065864    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 11:29:15.066286 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.066271    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:29:15.068979 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.068964    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 11:29:15.075792 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.075778    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:29:15.076619 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076607    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 11:29:15.076662 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076638    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 11:29:15.076800 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076790    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 11:29:15.076847 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076806    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 11:29:15.076847 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076813    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 11:29:15.076847 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076818    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 11:29:15.076847 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076828    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 11:29:15.076847 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.076834    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 11:29:15.077560 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.077550    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 11:29:15.077597 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.077561    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 11:29:15.077597 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.077569    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 11:29:15.077597 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.077575    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 11:29:15.077597 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.077589    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 11:29:15.077597 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.077598    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 11:29:15.079047 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.079035    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 11:29:15.079084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.079052    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 11:29:15.083235 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.083221    2579 server.go:1263] "Started kubelet"
Oct 30 11:29:15.083331 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.083309    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 11:29:15.083386 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.083334    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 11:29:15.083431 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.083413    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 11:29:15.087266 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.087029    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 11:29:15.087613 ip-10-0-142-105 systemd[1]: Started Kubernetes Kubelet.
Oct 30 11:29:15.090288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.090035    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 11:29:15.095692 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.095671    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 11:29:15.097401 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.097383    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 11:29:15.097401 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.097396    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 11:29:15.097571 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.097499    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.098164 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098144    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 11:29:15.098164 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098166    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 11:29:15.098287 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098143    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 11:29:15.098287 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098256    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 11:29:15.098287 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098285    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 11:29:15.098421 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098296    2579 factory.go:55] Registering systemd factory
Oct 30 11:29:15.098421 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098302    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 11:29:15.098421 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098313    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 11:29:15.098566 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098517    2579 factory.go:153] Registering CRI-O factory
Oct 30 11:29:15.098566 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098531    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 11:29:15.098566 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098549    2579 factory.go:103] Registering Raw factory
Oct 30 11:29:15.098566 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.098562    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 11:29:15.099665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.099647    2579 manager.go:319] Starting recovery of all containers
Oct 30 11:29:15.110069 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.110051    2579 manager.go:324] Recovery completed
Oct 30 11:29:15.114653 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.114640    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.116636 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.116609    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.116702 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.116650    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.116702 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.116663    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.117146 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117134    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 11:29:15.117204 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117147    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.117204 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117146    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 11:29:15.117204 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117183    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 11:29:15.117833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117821    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.117888 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117846    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.117888 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.117855    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.119366 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.119351    2579 policy_none.go:50] "Start"
Oct 30 11:29:15.119429 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.119371    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 11:29:15.119429 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.119386    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 11:29:15.120915 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.120893    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 11:29:15.121491 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.121479    2579 policy_none.go:44] "Start"
Oct 30 11:29:15.124549 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.124530    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 11:29:15.131318 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.131297    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 11:29:15.132411 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.131229    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18e34b3214343e8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-10-30 11:29:15.083005582 +0000 UTC m=+0.381627296,LastTimestamp:2026-10-30 11:29:15.083005582 +0000 UTC m=+0.381627296,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Oct 30 11:29:15.141058 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.140930    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 11:29:15.143129 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.143109    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2zszm"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.153433    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2zszm"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.165107    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.168985    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170100    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.170134    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170146    2579 server.go:87] "Starting device plugin registration server"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170350    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170368    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170373    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170379    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170396    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.170433    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170571    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170642    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.170650    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.171363    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.171409    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.174897    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.174914    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 11:29:15.176794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.174923    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18e34b3216356103  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-10-30 11:29:15.116634371 +0000 UTC m=+0.415256086,LastTimestamp:2026-10-30 11:29:15.116634371 +0000 UTC m=+0.415256086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Oct 30 11:29:15.194664 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.194648    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:15.270913 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.270848    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.271016 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.270850    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"]
Oct 30 11:29:15.271016 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271001    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.271813 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271789    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.271905 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271828    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.271905 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271832    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.271905 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271856    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.271905 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271866    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.271905 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271842    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.271905 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.271888    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.274084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274072    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.274229 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.274267 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274243    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.274789 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274771    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.274872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274771    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.274872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274815    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.274872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274825    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.274872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274833    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.274872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.274846    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.277551 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.277537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.277604 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.277561    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:15.278179 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.278163    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:15.278242 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.278193    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:15.278242 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.278207    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:15.295945 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.295923    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.295945 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.295944    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-105.ec2.internal\": node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.298538 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.298523    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-105.ec2.internal\" not found" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.307519 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.307500    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-105.ec2.internal\" not found" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.334879 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.334856    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.399196 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.399170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/148cbd5f3b955e9e79f6d6f208c89d56-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"148cbd5f3b955e9e79f6d6f208c89d56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.399196 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.399197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.399361 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.399213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.435286 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.435250    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.500107 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.500084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.500182 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.500111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.500182 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.500129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/148cbd5f3b955e9e79f6d6f208c89d56-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"148cbd5f3b955e9e79f6d6f208c89d56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.500182 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.500178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/148cbd5f3b955e9e79f6d6f208c89d56-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"148cbd5f3b955e9e79f6d6f208c89d56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.500287 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.500193    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.500287 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.500193    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.536167 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.536117    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.603947 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.603923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.612065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:15.612045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:15.637228 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.637202    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.737712 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.737671    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.838284 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.838200    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:15.938695 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:15.938665    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.018169 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.018141    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 11:29:16.018636 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.018272    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 11:29:16.039494 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.039439    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.075967 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.075941    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:16.139672 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.139597    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.158012 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:16.157981    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148cbd5f3b955e9e79f6d6f208c89d56.slice/crio-0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a WatchSource:0}: Error finding container 0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a: Status 404 returned error can't find the container with id 0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a
Oct 30 11:29:16.158424 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:16.158397    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1793a91293e97f2ab4b51edb090789a9.slice/crio-43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7 WatchSource:0}: Error finding container 43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7: Status 404 returned error can't find the container with id 43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7
Oct 30 11:29:16.160793 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.160762    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:24:15 +0000 UTC" deadline="2028-05-24 09:21:42.656949276 +0000 UTC"
Oct 30 11:29:16.160868 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.160795    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13725h52m26.496159498s"
Oct 30 11:29:16.165394 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.165380    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:29:16.174113 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.174079    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" event={"ID":"148cbd5f3b955e9e79f6d6f208c89d56","Type":"ContainerStarted","Data":"0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a"}
Oct 30 11:29:16.175059 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.175041    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"1793a91293e97f2ab4b51edb090789a9","Type":"ContainerStarted","Data":"43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7"}
Oct 30 11:29:16.240192 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.240150    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.277973 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.277935    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:16.333550 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.333515    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:16.334900 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.334881    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:16.334979 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.334916    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:16.334979 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.334927    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:16.341057 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.341036    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.442095 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.442017    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.542724 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.542685    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.643728 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:16.643693    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:16.650707 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.650670    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:16.697974 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.697948    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:16.725831 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.725802    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:16.727040 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.727018    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:16.749464 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:16.749422    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:17.066402 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.066317    2579 apiserver.go:52] "Watching apiserver"
Oct 30 11:29:17.098826 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.098795    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 11:29:17.110463 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.110434    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 11:29:17.110856 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.110833    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-58qds","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w","openshift-multus/multus-additional-cni-plugins-d2ml5","openshift-network-operator/iptables-alerter-vhm89","kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal","openshift-multus/network-metrics-daemon-6s7j4","openshift-network-diagnostics/network-check-target-vxtn4","openshift-ovn-kubernetes/ovnkube-node-n2zn7","openshift-dns/node-resolver-7lsq8","openshift-cluster-node-tuning-operator/tuned-64dlw","openshift-image-registry/node-ca-dhfxh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal","openshift-multus/multus-289w4","kube-system/global-pull-secret-syncer-r4xxg"]
Oct 30 11:29:17.113872 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.113852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:17.113972 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.113932    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:17.116276 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.116252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.118719 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.118696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.121243 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.121223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.123423 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.123401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.123547 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.123498    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:17.126398 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.126283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.126398 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.126357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:17.128966 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.128938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.132129 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.132098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.132643 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.132623    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.132739 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.132677    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.132739 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.132676    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmhtb\""
Oct 30 11:29:17.134259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.134079    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.134259 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.134137    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 11:29:17.134969 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.134555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.135075 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.135048    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hc7n5\""
Oct 30 11:29:17.137120 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.137092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.138969 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.138948    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.139074 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139056    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 11:29:17.139149 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139111    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.139149 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139136    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 11:29:17.139267 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139145    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.139387 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139371    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bltqf\""
Oct 30 11:29:17.139752 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139734    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 11:29:17.139832 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.139762    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 11:29:17.140129 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.140098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.143295 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.143268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.152151 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.152127    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlvg9\""
Oct 30 11:29:17.152261 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.152223    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 11:29:17.152261 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.152245    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.152578 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.152536    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.161436 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.161417    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 11:29:17.161545 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.161499    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cm97b"
Oct 30 11:29:17.199869 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.199857    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 11:29:17.210536 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-cni-binary-copy\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.210644 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-kubelet\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.210644 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwtx\" (UniqueName: \"kubernetes.io/projected/5339a2af-2f4a-4444-b146-dcb849fa7449-kube-api-access-hzwtx\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.210644 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-run\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.210789 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-lib-modules\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.210789 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-etc-kubernetes\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.210874 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.210874 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5wd\" (UniqueName: \"kubernetes.io/projected/fddeddb5-22c9-4d0b-a77f-6d2a12319856-kube-api-access-nj5wd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.210971 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-socket-dir-parent\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.210971 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-socket-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.210971 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-tmp\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.211103 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.210986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bzx\" (UniqueName: \"kubernetes.io/projected/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-kube-api-access-t9bzx\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.211103 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc72p\" (UniqueName: \"kubernetes.io/projected/e40c5726-14a1-4f26-8d5f-218b4ff946a8-kube-api-access-lc72p\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.211197 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovn-node-metrics-cert\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211197 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-system-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.211288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-bin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.211288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-etc-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:17.211413 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.211413 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-conf-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.211550 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-iptables-alerter-script\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.211550 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-netd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211647 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-hostroot\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.211647 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-sys\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.211647 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.211761 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-var-lib-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211761 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-netns\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.211761 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-daemon-config\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.211761 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-kubelet\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211928 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-systemd-units\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211928 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-ovn\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211928 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-slash\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.211928 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.211928 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5815b6d3-0bdd-45b2-a673-5f493146c0f8-tmp-dir\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.212105 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-k8s-cni-cncf-io\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.212105 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.211966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-registration-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.212105 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-sys-fs\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.212105 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-modprobe-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212105 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-conf\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec657dda-01bf-42d4-96de-1f9bb920dfff-konnectivity-ca\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-host-slash\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-systemd\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbt8b\" (UniqueName: \"kubernetes.io/projected/14ba76bf-37ab-43fd-a583-5a9343e75964-kube-api-access-tbt8b\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-log-socket\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.212302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-config\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-env-overrides\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5n6\" (UniqueName: \"kubernetes.io/projected/0f4370e6-112e-4d65-940f-5ca5263ce913-kube-api-access-gg5n6\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-multus-certs\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-device-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec657dda-01bf-42d4-96de-1f9bb920dfff-agent-certs\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46099da2-dc06-45b7-bc3c-139b84c896e0-serviceca\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.212665 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-host\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7jdn\" (UniqueName: \"kubernetes.io/projected/46099da2-dc06-45b7-bc3c-139b84c896e0-kube-api-access-v7jdn\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-system-cni-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-kubelet-config\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-multus\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-tuned\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-netns\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5815b6d3-0bdd-45b2-a673-5f493146c0f8-hosts-file\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.212953 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46099da2-dc06-45b7-bc3c-139b84c896e0-host\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.212982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-systemd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-cnibin\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-dbus\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-cnibin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhn7\" (UniqueName: \"kubernetes.io/projected/91cfe548-7bc2-4679-99a3-7212bdc92193-kube-api-access-zdhn7\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysconfig\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-var-lib-kubelet\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-node-log\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrt9\" (UniqueName: \"kubernetes.io/projected/5815b6d3-0bdd-45b2-a673-5f493146c0f8-kube-api-access-jbrt9\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.213288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-kubernetes\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.213762 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-bin\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.213762 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-script-lib\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.213762 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.213489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.241647 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.241589    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 11:29:17.241942 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.241915    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.241968    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.241972    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242001    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242024    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242038    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dtb5\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242068    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242072    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242079    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 11:29:17.242095 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242079    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hnpqt\""
Oct 30 11:29:17.242565 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242027    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 11:29:17.242565 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242076    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 11:29:17.242565 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242195    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 11:29:17.242565 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242208    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k7ct5\""
Oct 30 11:29:17.242565 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.242308    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xq7t5\""
Oct 30 11:29:17.248617 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.248578    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 11:29:17.248617 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.248579    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wql9s\""
Oct 30 11:29:17.314398 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-conf-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-iptables-alerter-script\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-netd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-hostroot\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-conf-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.314609 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-sys\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314634    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-netd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314640    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-var-lib-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-netns\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-daemon-config\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314719    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-sys\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-kubelet\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314742    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-var-lib-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314775    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-hostroot\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-kubelet\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-systemd-units\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.314833 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314832    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-netns\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-ovn\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.314853    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-ovn\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-slash\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-systemd-units\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5815b6d3-0bdd-45b2-a673-5f493146c0f8-tmp-dir\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.314964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:17.814941285 +0000 UTC m=+3.113562988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-k8s-cni-cncf-io\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-registration-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-sys-fs\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-modprobe-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-conf\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec657dda-01bf-42d4-96de-1f9bb920dfff-konnectivity-ca\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-host-slash\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5815b6d3-0bdd-45b2-a673-5f493146c0f8-tmp-dir\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.315202 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-systemd\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbt8b\" (UniqueName: \"kubernetes.io/projected/14ba76bf-37ab-43fd-a583-5a9343e75964-kube-api-access-tbt8b\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-systemd\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315276    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-iptables-alerter-script\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-log-socket\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-config\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-env-overrides\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315377    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-sys-fs\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.314963    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-slash\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5n6\" (UniqueName: \"kubernetes.io/projected/0f4370e6-112e-4d65-940f-5ca5263ce913-kube-api-access-gg5n6\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-daemon-config\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315594    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-modprobe-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315639    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-log-socket\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315740    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-conf\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.315766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-registration-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315782    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-host-slash\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-k8s-cni-cncf-io\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.315981    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-multus-certs\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-device-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec657dda-01bf-42d4-96de-1f9bb920dfff-agent-certs\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46099da2-dc06-45b7-bc3c-139b84c896e0-serviceca\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-host\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316195    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-env-overrides\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316201    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-config\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316206    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec657dda-01bf-42d4-96de-1f9bb920dfff-konnectivity-ca\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jdn\" (UniqueName: \"kubernetes.io/projected/46099da2-dc06-45b7-bc3c-139b84c896e0-kube-api-access-v7jdn\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.316251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316241    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-system-cni-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316287    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-device-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-multus-certs\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-kubelet-config\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-multus\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316340    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-tuned\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-netns\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5815b6d3-0bdd-45b2-a673-5f493146c0f8-hosts-file\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46099da2-dc06-45b7-bc3c-139b84c896e0-host\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-systemd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-cnibin\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46099da2-dc06-45b7-bc3c-139b84c896e0-serviceca\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-dbus\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.316724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-cnibin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316658    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhn7\" (UniqueName: \"kubernetes.io/projected/91cfe548-7bc2-4679-99a3-7212bdc92193-kube-api-access-zdhn7\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316682    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316714    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5815b6d3-0bdd-45b2-a673-5f493146c0f8-hosts-file\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316740    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysconfig\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316790    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46099da2-dc06-45b7-bc3c-139b84c896e0-host\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316423    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-host\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-kubelet-config\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316820    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-dbus\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316847    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-cnibin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316873    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-systemd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316881    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-multus\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-cnibin\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-netns\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317002    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-system-cni-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.317397 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.316703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysconfig\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-var-lib-kubelet\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-node-log\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317127    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-var-lib-kubelet\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrt9\" (UniqueName: \"kubernetes.io/projected/5815b6d3-0bdd-45b2-a673-5f493146c0f8-kube-api-access-jbrt9\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-kubernetes\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-node-log\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-bin\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.317216    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-script-lib\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317233    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-bin\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.317266    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret podName:63257d8b-a976-4db1-8721-98f7d7fefed8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:17.817248188 +0000 UTC m=+3.115869898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret") pod "global-pull-secret-syncer-r4xxg" (UID: "63257d8b-a976-4db1-8721-98f7d7fefed8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317274    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-cni-binary-copy\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317332    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-kubernetes\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.318354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-kubelet\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwtx\" (UniqueName: \"kubernetes.io/projected/5339a2af-2f4a-4444-b146-dcb849fa7449-kube-api-access-hzwtx\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-run\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-lib-modules\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-etc-kubernetes\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5wd\" (UniqueName: \"kubernetes.io/projected/fddeddb5-22c9-4d0b-a77f-6d2a12319856-kube-api-access-nj5wd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-socket-dir-parent\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-socket-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-tmp\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bzx\" (UniqueName: \"kubernetes.io/projected/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-kube-api-access-t9bzx\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc72p\" (UniqueName: \"kubernetes.io/projected/e40c5726-14a1-4f26-8d5f-218b4ff946a8-kube-api-access-lc72p\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovn-node-metrics-cert\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317744    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-system-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317749    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-script-lib\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.319238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-bin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-etc-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-system-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317878    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-lib-modules\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317932    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-socket-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317440    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-kubelet\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-cni-binary-copy\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.317967    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-etc-kubernetes\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.318012    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-bin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.318181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-etc-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.318242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-run\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.320124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.318460    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-socket-dir-parent\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.321890 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.321865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-tmp\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.321998 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.321974    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovn-node-metrics-cert\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.322185 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.322160    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec657dda-01bf-42d4-96de-1f9bb920dfff-agent-certs\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.322284 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.322258    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-tuned\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.338610 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.338582    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:17.338610 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.338615    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:17.338803 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.338630    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mpbfg for pod openshift-network-diagnostics/network-check-target-vxtn4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:17.338803 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.338702    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg podName:cf8a23a0-a643-42bd-84f7-29a2bb9138b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:17.838681881 +0000 UTC m=+3.137303616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mpbfg" (UniqueName: "kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg") pod "network-check-target-vxtn4" (UID: "cf8a23a0-a643-42bd-84f7-29a2bb9138b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:17.341886 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.341856    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbt8b\" (UniqueName: \"kubernetes.io/projected/14ba76bf-37ab-43fd-a583-5a9343e75964-kube-api-access-tbt8b\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.343751 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.343724    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrt9\" (UniqueName: \"kubernetes.io/projected/5815b6d3-0bdd-45b2-a673-5f493146c0f8-kube-api-access-jbrt9\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.343873 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.343793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5n6\" (UniqueName: \"kubernetes.io/projected/0f4370e6-112e-4d65-940f-5ca5263ce913-kube-api-access-gg5n6\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.346128 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.346102    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwtx\" (UniqueName: \"kubernetes.io/projected/5339a2af-2f4a-4444-b146-dcb849fa7449-kube-api-access-hzwtx\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.346128 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.346119    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bzx\" (UniqueName: \"kubernetes.io/projected/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-kube-api-access-t9bzx\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.346295 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.346130    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhn7\" (UniqueName: \"kubernetes.io/projected/91cfe548-7bc2-4679-99a3-7212bdc92193-kube-api-access-zdhn7\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.346355 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.346328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc72p\" (UniqueName: \"kubernetes.io/projected/e40c5726-14a1-4f26-8d5f-218b4ff946a8-kube-api-access-lc72p\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.348278 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.348239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5wd\" (UniqueName: \"kubernetes.io/projected/fddeddb5-22c9-4d0b-a77f-6d2a12319856-kube-api-access-nj5wd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.350690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.350645    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jdn\" (UniqueName: \"kubernetes.io/projected/46099da2-dc06-45b7-bc3c-139b84c896e0-kube-api-access-v7jdn\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.431506 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.431474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:17.441871 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.441841    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46099da2_dc06_45b7_bc3c_139b84c896e0.slice/crio-8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1 WatchSource:0}: Error finding container 8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1: Status 404 returned error can't find the container with id 8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1
Oct 30 11:29:17.454721 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.454696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:17.458949 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.458929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:17.463826 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.463805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:17.474673 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.474655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-289w4"
Oct 30 11:29:17.486692 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.486669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:17.490278 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.490254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:17.490939 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.490917    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4370e6_112e_4d65_940f_5ca5263ce913.slice/crio-eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091 WatchSource:0}: Error finding container eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091: Status 404 returned error can't find the container with id eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091
Oct 30 11:29:17.492089 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.491978    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5815b6d3_0bdd_45b2_a673_5f493146c0f8.slice/crio-fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353 WatchSource:0}: Error finding container fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353: Status 404 returned error can't find the container with id fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353
Oct 30 11:29:17.497813 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.497771    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1af9bdc_dd7e_4b70_9fb0_6a0a17b58225.slice/crio-9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87 WatchSource:0}: Error finding container 9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87: Status 404 returned error can't find the container with id 9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87
Oct 30 11:29:17.498430 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.498413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:17.505738 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.505714    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cfe548_7bc2_4679_99a3_7212bdc92193.slice/crio-d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8 WatchSource:0}: Error finding container d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8: Status 404 returned error can't find the container with id d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8
Oct 30 11:29:17.506917 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.506894    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ba76bf_37ab_43fd_a583_5a9343e75964.slice/crio-9ac204aaef24a91f9cbbb50bfba1591f9d8ca3c933433512eab8e897f40e7aa7 WatchSource:0}: Error finding container 9ac204aaef24a91f9cbbb50bfba1591f9d8ca3c933433512eab8e897f40e7aa7: Status 404 returned error can't find the container with id 9ac204aaef24a91f9cbbb50bfba1591f9d8ca3c933433512eab8e897f40e7aa7
Oct 30 11:29:17.514862 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.514836    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec657dda_01bf_42d4_96de_1f9bb920dfff.slice/crio-b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d WatchSource:0}: Error finding container b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d: Status 404 returned error can't find the container with id b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d
Oct 30 11:29:17.522342 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.522220    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:17.535180 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:17.535152    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddeddb5_22c9_4d0b_a77f_6d2a12319856.slice/crio-a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f WatchSource:0}: Error finding container a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f: Status 404 returned error can't find the container with id a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f
Oct 30 11:29:17.581108 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.581033    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:17.820771 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.820734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:17.820902 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.820825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:17.820902 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.820878    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:17.821006 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.820936    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:17.821006 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.820959    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret podName:63257d8b-a976-4db1-8721-98f7d7fefed8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:18.820936403 +0000 UTC m=+4.119558107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret") pod "global-pull-secret-syncer-r4xxg" (UID: "63257d8b-a976-4db1-8721-98f7d7fefed8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:17.821006 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.820992    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:18.82097449 +0000 UTC m=+4.119596205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:17.922021 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:17.921923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:17.922184 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.922124    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:17.922184 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.922145    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:17.922184 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.922160    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mpbfg for pod openshift-network-diagnostics/network-check-target-vxtn4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:17.922365 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:17.922302    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg podName:cf8a23a0-a643-42bd-84f7-29a2bb9138b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:18.922280849 +0000 UTC m=+4.220902552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpbfg" (UniqueName: "kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg") pod "network-check-target-vxtn4" (UID: "cf8a23a0-a643-42bd-84f7-29a2bb9138b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:18.165225 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.165154    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:24:17 +0000 UTC" deadline="2028-06-23 14:47:26.31446816 +0000 UTC"
Oct 30 11:29:18.165225 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.165222    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14451h18m8.14925057s"
Oct 30 11:29:18.195647 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.195609    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" event={"ID":"148cbd5f3b955e9e79f6d6f208c89d56","Type":"ContainerStarted","Data":"834ea26f3c0ced8af9b10a7cb6ba6597a78b92b4586251b181660ea8b2da710c"}
Oct 30 11:29:18.200427 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.200394    2579 generic.go:356] "Generic (PLEG): container finished" podID="1793a91293e97f2ab4b51edb090789a9" containerID="936382754aeb57f1d7d23f2eb4525fde9c54c8f70b1eecc4b8a2db3d144630bf" exitCode=0
Oct 30 11:29:18.200604 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.200483    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"1793a91293e97f2ab4b51edb090789a9","Type":"ContainerDied","Data":"936382754aeb57f1d7d23f2eb4525fde9c54c8f70b1eecc4b8a2db3d144630bf"}
Oct 30 11:29:18.214402 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.214358    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f"}
Oct 30 11:29:18.221814 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.221680    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-58qds" event={"ID":"ec657dda-01bf-42d4-96de-1f9bb920dfff","Type":"ContainerStarted","Data":"b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d"}
Oct 30 11:29:18.225422 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.225392    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-64dlw" event={"ID":"14ba76bf-37ab-43fd-a583-5a9343e75964","Type":"ContainerStarted","Data":"9ac204aaef24a91f9cbbb50bfba1591f9d8ca3c933433512eab8e897f40e7aa7"}
Oct 30 11:29:18.237789 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.237759    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w" event={"ID":"91cfe548-7bc2-4679-99a3-7212bdc92193","Type":"ContainerStarted","Data":"d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8"}
Oct 30 11:29:18.263980 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.263942    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-289w4" event={"ID":"5339a2af-2f4a-4444-b146-dcb849fa7449","Type":"ContainerStarted","Data":"1255d8a23fb23bd450dcafa6722e054f47a68521a7ac6c8e915cd6edf30c546d"}
Oct 30 11:29:18.277187 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.277156    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerStarted","Data":"eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091"}
Oct 30 11:29:18.285495 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.284721    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" podStartSLOduration=1.2847036649999999 podStartE2EDuration="1.284703665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:18.240679299 +0000 UTC m=+3.539301028" watchObservedRunningTime="2026-10-30 11:29:18.284703665 +0000 UTC m=+3.583325394"
Oct 30 11:29:18.290342 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.290312    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhm89" event={"ID":"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225","Type":"ContainerStarted","Data":"9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87"}
Oct 30 11:29:18.292314 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.292285    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7lsq8" event={"ID":"5815b6d3-0bdd-45b2-a673-5f493146c0f8","Type":"ContainerStarted","Data":"fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353"}
Oct 30 11:29:18.295709 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.295677    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dhfxh" event={"ID":"46099da2-dc06-45b7-bc3c-139b84c896e0","Type":"ContainerStarted","Data":"8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1"}
Oct 30 11:29:18.835293 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.834486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:18.835293 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.834562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:18.835293 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.834703    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:18.835293 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.834766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret podName:63257d8b-a976-4db1-8721-98f7d7fefed8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:20.834748135 +0000 UTC m=+6.133369866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret") pod "global-pull-secret-syncer-r4xxg" (UID: "63257d8b-a976-4db1-8721-98f7d7fefed8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:18.835293 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.835158    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:18.835293 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.835210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:20.835194518 +0000 UTC m=+6.133816236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:18.936480 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:18.935862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:18.936480 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.936039    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:18.936480 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.936060    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:18.936480 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.936072    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mpbfg for pod openshift-network-diagnostics/network-check-target-vxtn4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:18.936480 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:18.936129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg podName:cf8a23a0-a643-42bd-84f7-29a2bb9138b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:20.936111869 +0000 UTC m=+6.234733588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpbfg" (UniqueName: "kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg") pod "network-check-target-vxtn4" (UID: "cf8a23a0-a643-42bd-84f7-29a2bb9138b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:19.166127 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:19.165975    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:24:17 +0000 UTC" deadline="2028-04-05 18:23:41.024022183 +0000 UTC"
Oct 30 11:29:19.166127 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:19.166016    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12558h54m21.858011107s"
Oct 30 11:29:19.174194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:19.173702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:19.174194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:19.173726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:19.174194 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:19.173835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:19.174194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:19.173897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:19.174194 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:19.174007    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:19.174194 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:19.174109    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:19.315430 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:19.315343    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"1793a91293e97f2ab4b51edb090789a9","Type":"ContainerStarted","Data":"c2ef2193b6dcd8a1995bab13164647f8764810e309d25978d6989881c49381ff"}
Oct 30 11:29:20.852218 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:20.852178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:20.852766 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:20.852268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:20.852766 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.852412    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:20.852766 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.852473    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:24.852457847 +0000 UTC m=+10.151079569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:20.852942 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.852806    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:20.852942 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.852854    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret podName:63257d8b-a976-4db1-8721-98f7d7fefed8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:24.852837819 +0000 UTC m=+10.151459539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret") pod "global-pull-secret-syncer-r4xxg" (UID: "63257d8b-a976-4db1-8721-98f7d7fefed8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:20.953050 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:20.953010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:20.953341 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.953290    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:20.953421 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.953350    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:20.953421 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.953365    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mpbfg for pod openshift-network-diagnostics/network-check-target-vxtn4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:20.953563 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:20.953434    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg podName:cf8a23a0-a643-42bd-84f7-29a2bb9138b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:24.953414324 +0000 UTC m=+10.252036031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpbfg" (UniqueName: "kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg") pod "network-check-target-vxtn4" (UID: "cf8a23a0-a643-42bd-84f7-29a2bb9138b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:21.173319 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:21.172657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:21.173319 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:21.172799    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:21.173319 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:21.172816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:21.173319 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:21.172896    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:21.173319 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:21.172657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:21.173319 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:21.173002    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:21.333942 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:21.333127    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="a9d950ab1c27de249032c3e5b21cccc71b1144194fd6e70661e9185d5d585bf8" exitCode=0
Oct 30 11:29:21.333942 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:21.333190    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerDied","Data":"a9d950ab1c27de249032c3e5b21cccc71b1144194fd6e70661e9185d5d585bf8"}
Oct 30 11:29:21.410074 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:21.410006    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" podStartSLOduration=4.409985002 podStartE2EDuration="4.409985002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:19.358696331 +0000 UTC m=+4.657318056" watchObservedRunningTime="2026-10-30 11:29:21.409985002 +0000 UTC m=+6.708606732"
Oct 30 11:29:22.341291 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:22.341243    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-58qds" event={"ID":"ec657dda-01bf-42d4-96de-1f9bb920dfff","Type":"ContainerStarted","Data":"bbd4112e8668f58bcae77b6c56b994fb0bf301e39fe88dd098df2990cf4b2839"}
Oct 30 11:29:23.171355 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:23.171313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:23.171566 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:23.171313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:23.171566 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:23.171468    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:23.171690 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:23.171559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:23.171690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:23.171320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:23.171690 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:23.171649    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:23.346691 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:23.346611    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dhfxh" event={"ID":"46099da2-dc06-45b7-bc3c-139b84c896e0","Type":"ContainerStarted","Data":"8ddb9ea7b6ca28754b85f322ee80c2a263e80c591acce610148ddf982515a9d3"}
Oct 30 11:29:23.391723 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:23.390814    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-58qds" podStartSLOduration=3.437442185 podStartE2EDuration="7.390798248s" totalImagesPullingTime="3.953356063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:22.391570419 +0000 UTC m=+7.690192149" watchObservedRunningTime="2026-10-30 11:29:23.390798248 +0000 UTC m=+8.689419975"
Oct 30 11:29:24.352448 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:24.352352    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w" event={"ID":"91cfe548-7bc2-4679-99a3-7212bdc92193","Type":"ContainerStarted","Data":"48592640fee43b115e2898eb002cdc2f6fa0eca6f495eb268b8da9ff852ce9c5"}
Oct 30 11:29:24.888171 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:24.887506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:24.888171 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:24.887599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:24.888171 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.887716    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:24.888171 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.887716    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:24.888171 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.887786    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:32.887767943 +0000 UTC m=+18.186389663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:24.888171 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.887804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret podName:63257d8b-a976-4db1-8721-98f7d7fefed8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:32.887794432 +0000 UTC m=+18.186416136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret") pod "global-pull-secret-syncer-r4xxg" (UID: "63257d8b-a976-4db1-8721-98f7d7fefed8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:24.988728 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:24.988065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:24.988728 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.988225    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:24.988728 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.988245    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:24.988728 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.988256    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mpbfg for pod openshift-network-diagnostics/network-check-target-vxtn4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:24.988728 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:24.988316    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg podName:cf8a23a0-a643-42bd-84f7-29a2bb9138b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:32.988299094 +0000 UTC m=+18.286920813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpbfg" (UniqueName: "kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg") pod "network-check-target-vxtn4" (UID: "cf8a23a0-a643-42bd-84f7-29a2bb9138b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:25.084316 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.084162    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:25.084779 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.084755    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:25.128639 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.127890    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dhfxh" podStartSLOduration=4.177066204 podStartE2EDuration="9.127868729s" totalImagesPullingTime="4.950802525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:23.390796859 +0000 UTC m=+8.689418587" watchObservedRunningTime="2026-10-30 11:29:25.127868729 +0000 UTC m=+10.426490459"
Oct 30 11:29:25.176312 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.171876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:25.176312 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:25.171993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:25.176312 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.172391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:25.176312 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:25.172477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:25.176312 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.172554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:25.176312 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:25.172649    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:25.198257 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.198216    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 11:29:25.356596 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:25.356563    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w" event={"ID":"91cfe548-7bc2-4679-99a3-7212bdc92193","Type":"ContainerStarted","Data":"d3bde786440973b50780fc7498166c9e4c36f6689e69b08bf644f001d264679c"}
Oct 30 11:29:26.177653 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.177535    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:29:25.198235545Z","UUID":"52c5a5dd-6ae1-4c2b-949f-bbfa7a773bda","Handler":null,"Name":"","Endpoint":""}
Oct 30 11:29:26.179493 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.179278    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 11:29:26.179632 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.179510    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 11:29:26.361579 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.361294    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="b180b1661dc4482ee851b121f37bd64741aac7347e27b97b82c2ea470598fcaa" exitCode=0
Oct 30 11:29:26.362039 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.361336    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerDied","Data":"b180b1661dc4482ee851b121f37bd64741aac7347e27b97b82c2ea470598fcaa"}
Oct 30 11:29:26.363946 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.363828    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7lsq8" event={"ID":"5815b6d3-0bdd-45b2-a673-5f493146c0f8","Type":"ContainerStarted","Data":"d6def017833b4ffded95541f4c488b0ab6a633a23f34e602f282b1ca62eaa70f"}
Oct 30 11:29:26.365710 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.365687    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-289w4" event={"ID":"5339a2af-2f4a-4444-b146-dcb849fa7449","Type":"ContainerStarted","Data":"d274b0e040c2a1a538e76b146a93f232d270c06872b9a3bb55f88bf168cc8e28"}
Oct 30 11:29:26.460212 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.459934    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-289w4" podStartSLOduration=2.477291187 podStartE2EDuration="10.459912069s" totalImagesPullingTime="7.982620882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:26.459217115 +0000 UTC m=+11.757838825" watchObservedRunningTime="2026-10-30 11:29:26.459912069 +0000 UTC m=+11.758533797"
Oct 30 11:29:26.501474 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:26.501398    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7lsq8" podStartSLOduration=1.95874232 podStartE2EDuration="10.501382256s" totalImagesPullingTime="8.542639936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:26.501252156 +0000 UTC m=+11.799873882" watchObservedRunningTime="2026-10-30 11:29:26.501382256 +0000 UTC m=+11.800003983"
Oct 30 11:29:27.171504 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.171453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:27.171696 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.171453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:27.171696 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:27.171640    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:27.171696 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:27.171668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:27.171835 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.171812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:27.171920 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:27.171901    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:27.371288 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.371245    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="8e051263ba5ddaa03513ee31c2b31afbaf37532efac55cc8d78374e2e78a9916" exitCode=0
Oct 30 11:29:27.371883 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.371858    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerDied","Data":"8e051263ba5ddaa03513ee31c2b31afbaf37532efac55cc8d78374e2e78a9916"}
Oct 30 11:29:27.374543 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.374512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhm89" event={"ID":"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225","Type":"ContainerStarted","Data":"35922b97ded86849fffe3b22b1d4374a2952c67dcebd95ad21dace545998573d"}
Oct 30 11:29:27.378430 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.378393    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w" event={"ID":"91cfe548-7bc2-4679-99a3-7212bdc92193","Type":"ContainerStarted","Data":"5d33223c29cedc47d17b788cdc61205281e9429621dac2dd27b88d39ee4cff64"}
Oct 30 11:29:27.457535 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.457487    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vhm89" podStartSLOduration=3.467083182 podStartE2EDuration="11.4574718s" totalImagesPullingTime="7.990388618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:27.457416888 +0000 UTC m=+12.756038608" watchObservedRunningTime="2026-10-30 11:29:27.4574718 +0000 UTC m=+12.756093525"
Oct 30 11:29:27.499968 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:27.499919    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w" podStartSLOduration=3.006004287 podStartE2EDuration="11.499905418s" totalImagesPullingTime="8.493901131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:27.499900504 +0000 UTC m=+12.798522230" watchObservedRunningTime="2026-10-30 11:29:27.499905418 +0000 UTC m=+12.798527143"
Oct 30 11:29:28.382587 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:28.382550    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-64dlw" event={"ID":"14ba76bf-37ab-43fd-a583-5a9343e75964","Type":"ContainerStarted","Data":"f6eb53c484de54ef42720e4e6ca50feedbdce845153bade951230a080a71600a"}
Oct 30 11:29:28.385194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:28.385168    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="d5c2b484b4d6e11c22abbe557dfd7026a4e2123e1e655cde85a11192c7df6c40" exitCode=0
Oct 30 11:29:28.385328 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:28.385250    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerDied","Data":"d5c2b484b4d6e11c22abbe557dfd7026a4e2123e1e655cde85a11192c7df6c40"}
Oct 30 11:29:28.425788 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:28.425735    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-64dlw" podStartSLOduration=1.7221943 podStartE2EDuration="12.425723017s" totalImagesPullingTime="10.703528717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:28.42516465 +0000 UTC m=+13.723786373" watchObservedRunningTime="2026-10-30 11:29:28.425723017 +0000 UTC m=+13.724344755"
Oct 30 11:29:29.170727 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:29.170688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:29.170910 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:29.170813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:29.171376 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:29.171357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:29.171492 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:29.171441    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:29.171550 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:29.171525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:29.171660 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:29.171645    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:31.171215 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:31.170971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:31.171918 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:31.170975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:31.171918 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:31.171038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:31.171918 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:31.171330    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:31.171918 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:31.171387    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:31.171918 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:31.171491    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:31.394944 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:31.394901    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="9acaeaf4009561452d897851adfed5a5a14b5fed37c14c64057505484bf9a7be" exitCode=0
Oct 30 11:29:31.395096 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:31.394979    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerDied","Data":"9acaeaf4009561452d897851adfed5a5a14b5fed37c14c64057505484bf9a7be"}
Oct 30 11:29:32.400599 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:32.400560    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="8afefd9c6a9d2a5799e281eccb81e22faae141a9bef698cada7e237034766920" exitCode=0
Oct 30 11:29:32.401096 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:32.400605    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerDied","Data":"8afefd9c6a9d2a5799e281eccb81e22faae141a9bef698cada7e237034766920"}
Oct 30 11:29:32.949036 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:32.948855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:32.949219 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:32.949075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:32.949219 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:32.949012    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:32.949219 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:32.949153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret podName:63257d8b-a976-4db1-8721-98f7d7fefed8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.949138052 +0000 UTC m=+34.247759754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret") pod "global-pull-secret-syncer-r4xxg" (UID: "63257d8b-a976-4db1-8721-98f7d7fefed8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:32.949219 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:32.949190    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:32.949369 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:32.949247    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.949232065 +0000 UTC m=+34.247853767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:33.049687 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.049656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:33.049808 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.049778    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:33.049808 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.049792    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:33.049808 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.049802    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mpbfg for pod openshift-network-diagnostics/network-check-target-vxtn4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:33.049912 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.049849    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg podName:cf8a23a0-a643-42bd-84f7-29a2bb9138b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:49.049834061 +0000 UTC m=+34.348455763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpbfg" (UniqueName: "kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg") pod "network-check-target-vxtn4" (UID: "cf8a23a0-a643-42bd-84f7-29a2bb9138b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:33.170706 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.170670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:33.170865 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.170815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:33.170865 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.170827    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:33.170865 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.170843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:33.170952 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.170911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:33.170992 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:33.170978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:33.405694 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.405610    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" event={"ID":"0f4370e6-112e-4d65-940f-5ca5263ce913","Type":"ContainerStarted","Data":"c7ed1e410a31f46c24715839439a33306ae3fee393062f043e6a1bbc81815727"}
Oct 30 11:29:33.408464 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.408426    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"203e782184a602366e24b4d2e27e1f707306106963560f2fa42fa65c1e175e83"}
Oct 30 11:29:33.408596 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.408516    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"2e39aa5425c8271fcfb324eafa1bce2946e03a77e5cd2eed8078151de5332087"}
Oct 30 11:29:33.408596 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.408565    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"8a294564e8d5d8e75eda046d0878c7787d898c473b6ee68fd2b8712d2d06c9a1"}
Oct 30 11:29:33.408596 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.408595    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"bf3e54673f6755301c7ae5deb9ebbe73f390d6fed9b62246b8a5438a9c589a03"}
Oct 30 11:29:33.408744 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.408621    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"1eba23e964c1f3417d054ba5c92b55b0d5ded6ae2c197810cbc1b34fc99a4b9d"}
Oct 30 11:29:33.408744 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.408656    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"3cc37e58a86e8d7dce97b83b7d2392df7af7382aa7c9f9ec3b356c288e49f061"}
Oct 30 11:29:33.453976 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:33.453925    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d2ml5" podStartSLOduration=7.045402504 podStartE2EDuration="17.453912277s" totalImagesPullingTime="10.408509773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:33.453755372 +0000 UTC m=+18.752377095" watchObservedRunningTime="2026-10-30 11:29:33.453912277 +0000 UTC m=+18.752534000"
Oct 30 11:29:35.171613 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:35.171520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:35.172192 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:35.171644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:35.172192 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:35.171668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:35.172192 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:35.171712    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:35.172192 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:35.171749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:35.172192 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:35.171809    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:35.415852 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:35.415818    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"d64170ba5b4fef4cdb480c30bb264254372c1a8b766e3b16957d9fcb1ac70f96"}
Oct 30 11:29:37.167488 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.167378    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:37.168238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.167536    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:29:37.168238 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.168195    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:37.170724 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.170704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:37.170824 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.170718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:37.170824 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.170814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:37.170824 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:37.170811    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:37.170925 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:37.170882    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:37.170969 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:37.170951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:37.424014 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.423915    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" event={"ID":"fddeddb5-22c9-4d0b-a77f-6d2a12319856","Type":"ContainerStarted","Data":"5743bd79b077bee139277a315d89ad378cf4560141f0001457fcc565cec08513"}
Oct 30 11:29:37.489160 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:37.489109    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7" podStartSLOduration=6.5428226590000005 podStartE2EDuration="21.489096055s" totalImagesPullingTime="14.946273396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:37.489021977 +0000 UTC m=+22.787643723" watchObservedRunningTime="2026-10-30 11:29:37.489096055 +0000 UTC m=+22.787717804"
Oct 30 11:29:38.426589 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:38.426558    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:38.427050 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:38.426644    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:38.427050 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:38.426688    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:38.442411 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:38.442387    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:38.442560 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:38.442516    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:39.009677 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:39.009648    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r4xxg"]
Oct 30 11:29:39.009807 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:39.009758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:39.009864 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:39.009844    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:39.013235 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:39.013208    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxtn4"]
Oct 30 11:29:39.013353 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:39.013296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:39.013402 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:39.013382    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:39.016314 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:39.016293    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6s7j4"]
Oct 30 11:29:39.016420 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:39.016393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:39.019940 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:39.019911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:40.171085 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:40.170912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:40.171492 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:40.171153    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:41.171185 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:41.171155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:41.171573 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:41.171168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:41.171573 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:41.171301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:41.171573 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:41.171324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:42.171349 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:42.171314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:42.171885 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:42.171433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:43.170753 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:43.170673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:43.170911 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:43.170785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s7j4" podUID="e40c5726-14a1-4f26-8d5f-218b4ff946a8"
Oct 30 11:29:43.170911 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:43.170870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:43.171021 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:43.170971    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4xxg" podUID="63257d8b-a976-4db1-8721-98f7d7fefed8"
Oct 30 11:29:44.171393 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:44.171357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:44.171853 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:44.171481    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxtn4" podUID="cf8a23a0-a643-42bd-84f7-29a2bb9138b7"
Oct 30 11:29:44.965247 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:44.965220    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeReady"
Oct 30 11:29:44.965350 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:44.965335    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 11:29:45.022906 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.022873    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5b47m"]
Oct 30 11:29:45.026225 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.026201    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-488kk"]
Oct 30 11:29:45.026351 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.026332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:45.028952 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.028931    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x2vn2"]
Oct 30 11:29:45.029068 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.029051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:45.031704 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.031689    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d654c48b4-zcdvc"]
Oct 30 11:29:45.032026 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.032005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:45.034525 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.034507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.035321 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.035306    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-25xnv"]
Oct 30 11:29:45.038850 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.038833    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d8f76c5df-xxgbt"]
Oct 30 11:29:45.038965 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.038943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.041632 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.041615    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"]
Oct 30 11:29:45.041813 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.041746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.044801 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.044783    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"]
Oct 30 11:29:45.044913 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.044896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.047638 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.047620    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-scqxd"]
Oct 30 11:29:45.047742 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.047728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.050857 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.050838    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"]
Oct 30 11:29:45.051551 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.051535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.054129 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.054104    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.054232 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.054203    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.056231 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.056207    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"]
Oct 30 11:29:45.056969 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.056949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:45.059711 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.059694    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"]
Oct 30 11:29:45.059810 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.059795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.062385 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.062367    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"]
Oct 30 11:29:45.062524 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.062504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.064264 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.064247    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 11:29:45.064354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.064303    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.065317 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.065225    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"]
Oct 30 11:29:45.065434 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.065409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:45.069258 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.069184    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f7dc85469-gjnw7"]
Oct 30 11:29:45.070434 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.070414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.073600 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.073580    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zgz8g"]
Oct 30 11:29:45.073690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.073655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.076435 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.076421    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pnclc"]
Oct 30 11:29:45.076580 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.076564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.077944 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.077929    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 11:29:45.082148 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.082127    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5b47m"]
Oct 30 11:29:45.082247 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.082221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.089770 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.089740    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-chtrx\""
Oct 30 11:29:45.089952 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.089934    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 11:29:45.090083 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.090065    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 11:29:45.090203 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.089993    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 11:29:45.090305 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.089817    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 11:29:45.091867 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.091248    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"]
Oct 30 11:29:45.092347 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.092328    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-488kk"]
Oct 30 11:29:45.093109 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.093093    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d654c48b4-zcdvc"]
Oct 30 11:29:45.093370 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.093350    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"]
Oct 30 11:29:45.094417 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.094396    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d8f76c5df-xxgbt"]
Oct 30 11:29:45.095121 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.095101    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-scqxd"]
Oct 30 11:29:45.095737 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.095717    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-25xnv"]
Oct 30 11:29:45.096396 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.096374    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"]
Oct 30 11:29:45.096982 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.096967    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"]
Oct 30 11:29:45.097621 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.097605    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"]
Oct 30 11:29:45.109694 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.109408    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"]
Oct 30 11:29:45.111720 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.111662    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f7dc85469-gjnw7"]
Oct 30 11:29:45.117205 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.116639    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x2vn2"]
Oct 30 11:29:45.123225 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.123204    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pnclc"]
Oct 30 11:29:45.124713 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.124693    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"]
Oct 30 11:29:45.128331 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.128307    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgz8g"]
Oct 30 11:29:45.138030 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62l8\" (UniqueName: \"kubernetes.io/projected/a4ebbfa3-4272-4095-8a57-a55146be5135-kube-api-access-s62l8\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.138134 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-default-certificate\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.138134 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4ebbfa3-4272-4095-8a57-a55146be5135-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.138309 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23360850-74e3-473e-a579-70d4fcc5a9d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.138309 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-snapshots\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.138309 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cec9279a-2cd4-49bd-823c-398f6528bbe7-serving-cert\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.138309 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-config\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.138309 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-tmp\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.138578 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.138578 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.138578 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.138578 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4mfs\" (UniqueName: \"kubernetes.io/projected/52a6b071-d9c7-4d71-aeae-65e818f74dec-kube-api-access-q4mfs\") pod \"network-check-source-6f78d59f99-7jfl7\" (UID: \"52a6b071-d9c7-4d71-aeae-65e818f74dec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:45.138578 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.138794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcb6\" (UniqueName: \"kubernetes.io/projected/5c714f27-241c-4c7f-86fe-2c3eb89d0542-kube-api-access-bjcb6\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.138794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.138794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.138794 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5b8\" (UniqueName: \"kubernetes.io/projected/8c68e591-1f33-4482-9843-d0412cec6bce-kube-api-access-nr5b8\") pod \"kas-connection-checker-685895c7c9-488kk\" (UID: \"8c68e591-1f33-4482-9843-d0412cec6bce\") " pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:45.138914 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.138914 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.138914 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzlw\" (UniqueName: \"kubernetes.io/projected/5d1d8529-d285-41d3-b179-6f3343059567-kube-api-access-bmzlw\") pod \"volume-data-source-validator-58c9f5f46d-7mg8k\" (UID: \"5d1d8529-d285-41d3-b179-6f3343059567\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:45.139011 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.139011 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.139011 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm8x\" (UniqueName: \"kubernetes.io/projected/ea88fbd7-988d-44a5-a12e-fd5e64e8da79-kube-api-access-kpm8x\") pod \"kas-connection-checker-685895c7c9-x2vn2\" (UID: \"ea88fbd7-988d-44a5-a12e-fd5e64e8da79\") " pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:45.139011 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.138997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c173a778-6503-4ebc-9bd4-750a98b0f9d9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.139176 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzcb\" (UniqueName: \"kubernetes.io/projected/cec9279a-2cd4-49bd-823c-398f6528bbe7-kube-api-access-whzcb\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.139176 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.139176 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-trusted-ca\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.139176 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-stats-auth\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.139176 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfjn\" (UniqueName: \"kubernetes.io/projected/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-kube-api-access-mrfjn\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.139338 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvld\" (UniqueName: \"kubernetes.io/projected/23360850-74e3-473e-a579-70d4fcc5a9d0-kube-api-access-6gvld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.139338 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-service-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.139338 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23360850-74e3-473e-a579-70d4fcc5a9d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.139338 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.139476 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297ff\" (UniqueName: \"kubernetes.io/projected/e3746ecb-830d-4d86-8400-21f4d2985b22-kube-api-access-297ff\") pod \"kas-connection-checker-685895c7c9-5b47m\" (UID: \"e3746ecb-830d-4d86-8400-21f4d2985b22\") " pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:45.139476 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.139476 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-serving-cert\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.139476 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.139636 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvg6b\" (UniqueName: \"kubernetes.io/projected/1f5f4127-5c51-4029-8ba2-0f5c01bad573-kube-api-access-lvg6b\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.139636 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.139544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.149213 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149189    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149304 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149235    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149371 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149234    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149371 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149234    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 11:29:45.149371 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149246    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149371 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149280    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.149581 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149282    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.149581 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149394    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.149581 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149423    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149714 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149686    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149714 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149691    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.149804 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149712    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 11:29:45.149804 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149748    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 11:29:45.149804 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149750    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.149804 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149762    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.149804 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149783    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.149804 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149804    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149834    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149852    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149885    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149890    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149911    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.149996    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.150084 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.150026    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:45.150374 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.150090    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 11:29:45.150374 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.150220    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 11:29:45.150484 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.150403    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 11:29:45.150805 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.150787    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 11:29:45.154282 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154211    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 11:29:45.154282 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154229    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 11:29:45.154282 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154229    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 11:29:45.154282 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154232    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154336    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154347    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154349    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154364    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154486    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154507    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154519    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4s6wd\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154520    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hm9cg\""
Oct 30 11:29:45.154558 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154538    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 11:29:45.154991 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154626    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2726q\""
Oct 30 11:29:45.154991 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.154703    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 11:29:45.174637 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.174612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:45.175024 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.174774    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:45.240009 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.239972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvld\" (UniqueName: \"kubernetes.io/projected/23360850-74e3-473e-a579-70d4fcc5a9d0-kube-api-access-6gvld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.240009 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-service-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.240229 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23360850-74e3-473e-a579-70d4fcc5a9d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.240229 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.240229 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.240229 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297ff\" (UniqueName: \"kubernetes.io/projected/e3746ecb-830d-4d86-8400-21f4d2985b22-kube-api-access-297ff\") pod \"kas-connection-checker-685895c7c9-5b47m\" (UID: \"e3746ecb-830d-4d86-8400-21f4d2985b22\") " pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:45.240424 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.240424 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-serving-cert\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.240424 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.240424 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvg6b\" (UniqueName: \"kubernetes.io/projected/1f5f4127-5c51-4029-8ba2-0f5c01bad573-kube-api-access-lvg6b\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.240424 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22a93e48-3382-47af-adef-2ceb0d1fc6aa-cert\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62l8\" (UniqueName: \"kubernetes.io/projected/a4ebbfa3-4272-4095-8a57-a55146be5135-kube-api-access-s62l8\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-tmp-dir\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59qz\" (UniqueName: \"kubernetes.io/projected/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-kube-api-access-b59qz\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-default-certificate\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4ebbfa3-4272-4095-8a57-a55146be5135-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23360850-74e3-473e-a579-70d4fcc5a9d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-snapshots\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cec9279a-2cd4-49bd-823c-398f6528bbe7-serving-cert\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240779    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-service-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.240840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5013b2f-c423-4c36-aaa0-917c35cf23f8-serving-cert\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.240984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-config\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-tmp\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241088    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4mfs\" (UniqueName: \"kubernetes.io/projected/52a6b071-d9c7-4d71-aeae-65e818f74dec-kube-api-access-q4mfs\") pod \"network-check-source-6f78d59f99-7jfl7\" (UID: \"52a6b071-d9c7-4d71-aeae-65e818f74dec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcb6\" (UniqueName: \"kubernetes.io/projected/5c714f27-241c-4c7f-86fe-2c3eb89d0542-kube-api-access-bjcb6\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241291    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.241443 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.241407    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5b8\" (UniqueName: \"kubernetes.io/projected/8c68e591-1f33-4482-9843-d0412cec6bce-kube-api-access-nr5b8\") pod \"kas-connection-checker-685895c7c9-488kk\" (UID: \"8c68e591-1f33-4482-9843-d0412cec6bce\") " pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241605    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.241868    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.241912    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.741895968 +0000 UTC m=+31.040517690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.241978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-config\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242062    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4ebbfa3-4272-4095-8a57-a55146be5135-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.242184 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242104    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-snapshots\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.242605 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.242303    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.742264536 +0000 UTC m=+31.040886247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:45.242605 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.242605 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.242525    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:45.242605 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.242605 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.242570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.74255479 +0000 UTC m=+31.041176509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:45.242605 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242630    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23360850-74e3-473e-a579-70d4fcc5a9d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.242665    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.242678    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d654c48b4-zcdvc: secret "image-registry-tls" not found
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242708    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-tmp\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.242737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls podName:bebf069a-0c3e-44c6-9ad8-23dc0056c442 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.742724145 +0000 UTC m=+31.041345846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls") pod "image-registry-7d654c48b4-zcdvc" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442") : secret "image-registry-tls" not found
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5013b2f-c423-4c36-aaa0-917c35cf23f8-config\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzlw\" (UniqueName: \"kubernetes.io/projected/5d1d8529-d285-41d3-b179-6f3343059567-kube-api-access-bmzlw\") pod \"volume-data-source-validator-58c9f5f46d-7mg8k\" (UID: \"5d1d8529-d285-41d3-b179-6f3343059567\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.242983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.243061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.243098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.243166    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.243208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzgg\" (UniqueName: \"kubernetes.io/projected/22a93e48-3382-47af-adef-2ceb0d1fc6aa-kube-api-access-5wzgg\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.243395 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.243240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrh4\" (UniqueName: \"kubernetes.io/projected/c5013b2f-c423-4c36-aaa0-917c35cf23f8-kube-api-access-hrrh4\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.244362 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.243421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls podName:1f5f4127-5c51-4029-8ba2-0f5c01bad573 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.74340503 +0000 UTC m=+31.042026733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lwp9b" (UID: "1f5f4127-5c51-4029-8ba2-0f5c01bad573") : secret "samples-operator-tls" not found
Oct 30 11:29:45.244362 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.244170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm8x\" (UniqueName: \"kubernetes.io/projected/ea88fbd7-988d-44a5-a12e-fd5e64e8da79-kube-api-access-kpm8x\") pod \"kas-connection-checker-685895c7c9-x2vn2\" (UID: \"ea88fbd7-988d-44a5-a12e-fd5e64e8da79\") " pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:45.244362 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.244219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c173a778-6503-4ebc-9bd4-750a98b0f9d9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.244362 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.244252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-config-volume\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.244362 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.244306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.244362 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.244340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzcb\" (UniqueName: \"kubernetes.io/projected/cec9279a-2cd4-49bd-823c-398f6528bbe7-kube-api-access-whzcb\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.245243 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c173a778-6503-4ebc-9bd4-750a98b0f9d9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-trusted-ca\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.245551    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.745537622 +0000 UTC m=+31.044159346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : configmap references non-existent config key: service-ca.crt
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-stats-auth\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245707    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-default-certificate\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfjn\" (UniqueName: \"kubernetes.io/projected/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-kube-api-access-mrfjn\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245806    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-serving-cert\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.246111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.245857    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23360850-74e3-473e-a579-70d4fcc5a9d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.246814 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.246310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.246814 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.246428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cec9279a-2cd4-49bd-823c-398f6528bbe7-serving-cert\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.247210 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.247186    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-trusted-ca\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.247708 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.247689    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.248069 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.248050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-stats-auth\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.261474 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.261431    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hsk2h\""
Oct 30 11:29:45.261568 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.261473    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 11:29:45.261687 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.261670    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 11:29:45.261722 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.261698    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-899rd\""
Oct 30 11:29:45.266194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.266169    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.267353 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.267334    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4mfs\" (UniqueName: \"kubernetes.io/projected/52a6b071-d9c7-4d71-aeae-65e818f74dec-kube-api-access-q4mfs\") pod \"network-check-source-6f78d59f99-7jfl7\" (UID: \"52a6b071-d9c7-4d71-aeae-65e818f74dec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:45.267689 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.267664    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcb6\" (UniqueName: \"kubernetes.io/projected/5c714f27-241c-4c7f-86fe-2c3eb89d0542-kube-api-access-bjcb6\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.268240 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.268220    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvg6b\" (UniqueName: \"kubernetes.io/projected/1f5f4127-5c51-4029-8ba2-0f5c01bad573-kube-api-access-lvg6b\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.270499 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.270478    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297ff\" (UniqueName: \"kubernetes.io/projected/e3746ecb-830d-4d86-8400-21f4d2985b22-kube-api-access-297ff\") pod \"kas-connection-checker-685895c7c9-5b47m\" (UID: \"e3746ecb-830d-4d86-8400-21f4d2985b22\") " pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:45.273926 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.273904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvld\" (UniqueName: \"kubernetes.io/projected/23360850-74e3-473e-a579-70d4fcc5a9d0-kube-api-access-6gvld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.280346 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.280326    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.281055 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.280363    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5b8\" (UniqueName: \"kubernetes.io/projected/8c68e591-1f33-4482-9843-d0412cec6bce-kube-api-access-nr5b8\") pod \"kas-connection-checker-685895c7c9-488kk\" (UID: \"8c68e591-1f33-4482-9843-d0412cec6bce\") " pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:45.282750 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.282719    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm8x\" (UniqueName: \"kubernetes.io/projected/ea88fbd7-988d-44a5-a12e-fd5e64e8da79-kube-api-access-kpm8x\") pod \"kas-connection-checker-685895c7c9-x2vn2\" (UID: \"ea88fbd7-988d-44a5-a12e-fd5e64e8da79\") " pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:45.283121 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.283097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfjn\" (UniqueName: \"kubernetes.io/projected/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-kube-api-access-mrfjn\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.283387 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.283368    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzlw\" (UniqueName: \"kubernetes.io/projected/5d1d8529-d285-41d3-b179-6f3343059567-kube-api-access-bmzlw\") pod \"volume-data-source-validator-58c9f5f46d-7mg8k\" (UID: \"5d1d8529-d285-41d3-b179-6f3343059567\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:45.288876 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.288849    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzcb\" (UniqueName: \"kubernetes.io/projected/cec9279a-2cd4-49bd-823c-398f6528bbe7-kube-api-access-whzcb\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.289435 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.289413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62l8\" (UniqueName: \"kubernetes.io/projected/a4ebbfa3-4272-4095-8a57-a55146be5135-kube-api-access-s62l8\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.336957 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.336918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:45.340838 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.340817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:45.346270 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:45.346486 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.346598 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5013b2f-c423-4c36-aaa0-917c35cf23f8-serving-cert\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.346598 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.346598 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.346785 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.346840 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5013b2f-c423-4c36-aaa0-917c35cf23f8-config\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.346892 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346831    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.346892 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.346988 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzgg\" (UniqueName: \"kubernetes.io/projected/22a93e48-3382-47af-adef-2ceb0d1fc6aa-kube-api-access-5wzgg\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.346988 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrh4\" (UniqueName: \"kubernetes.io/projected/c5013b2f-c423-4c36-aaa0-917c35cf23f8-kube-api-access-hrrh4\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.346988 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-config-volume\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.346988 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.346973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.347194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.347194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.347194 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347141    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.347332 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22a93e48-3382-47af-adef-2ceb0d1fc6aa-cert\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.347332 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-tmp-dir\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.347332 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59qz\" (UniqueName: \"kubernetes.io/projected/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-kube-api-access-b59qz\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.347332 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.347555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347496    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5013b2f-c423-4c36-aaa0-917c35cf23f8-config\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.348002 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.347980    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.348213 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.348196    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:45.348347 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.348335    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls podName:1d000fdc-e5ed-4fce-bb1e-6715932a37b1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.848315334 +0000 UTC m=+31.146937059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls") pod "dns-default-zgz8g" (UID: "1d000fdc-e5ed-4fce-bb1e-6715932a37b1") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:45.348441 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.348249    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:45.348546 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.348535    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f7dc85469-gjnw7: secret "image-registry-tls" not found
Oct 30 11:29:45.348677 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.348660    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls podName:d58c2f33-be41-4e16-91bc-293ee27d908e nodeName:}" failed. No retries permitted until 2026-10-30 11:29:45.848645235 +0000 UTC m=+31.147266972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls") pod "image-registry-7f7dc85469-gjnw7" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e") : secret "image-registry-tls" not found
Oct 30 11:29:45.348677 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.348659    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-tmp-dir\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.349421 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.349358    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-config-volume\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.350108 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.350076    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.350282 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.350250    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5013b2f-c423-4c36-aaa0-917c35cf23f8-serving-cert\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.350779 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.350757    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22a93e48-3382-47af-adef-2ceb0d1fc6aa-cert\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.350932 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.350900    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.357144 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.357118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:45.372216 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.372136    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrh4\" (UniqueName: \"kubernetes.io/projected/c5013b2f-c423-4c36-aaa0-917c35cf23f8-kube-api-access-hrrh4\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.373434 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.373408    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzgg\" (UniqueName: \"kubernetes.io/projected/22a93e48-3382-47af-adef-2ceb0d1fc6aa-kube-api-access-5wzgg\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.373603 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.373585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:45.375329 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.375303    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.375430 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.375374    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.379181 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.379156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:45.382880 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.382847    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59qz\" (UniqueName: \"kubernetes.io/projected/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-kube-api-access-b59qz\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.405332 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.405060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:45.419193 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.416500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:45.441576 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.441550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:45.456470 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.455574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:45.466939 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.466029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:45.574871 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.568266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:45.666148 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.666127    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5b47m"]
Oct 30 11:29:45.668643 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:45.668617    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3746ecb_830d_4d86_8400_21f4d2985b22.slice/crio-09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d WatchSource:0}: Error finding container 09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d: Status 404 returned error can't find the container with id 09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d
Oct 30 11:29:45.672831 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.672811    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-488kk"]
Oct 30 11:29:45.675978 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:45.675957    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c68e591_1f33_4482_9843_d0412cec6bce.slice/crio-f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6 WatchSource:0}: Error finding container f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6: Status 404 returned error can't find the container with id f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6
Oct 30 11:29:45.704513 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.703564    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x2vn2"]
Oct 30 11:29:45.747440 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.747405    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-scqxd"]
Oct 30 11:29:45.753378 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.753345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:45.753531 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.753392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.753531 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.753423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:45.753531 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.753482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:45.753531 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.753524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:45.753736 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.753563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:45.753794 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.753738    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.753719055 +0000 UTC m=+32.052340774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : configmap references non-existent config key: service-ca.crt
Oct 30 11:29:45.754106 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.753967    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:45.754106 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.753981    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:45.754106 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.754014151 +0000 UTC m=+32.052635875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:45.754106 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754051    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.754041302 +0000 UTC m=+32.052663007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:45.754106 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754077    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 11:29:45.754440 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754113    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls podName:1f5f4127-5c51-4029-8ba2-0f5c01bad573 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.754097985 +0000 UTC m=+32.052719719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lwp9b" (UID: "1f5f4127-5c51-4029-8ba2-0f5c01bad573") : secret "samples-operator-tls" not found
Oct 30 11:29:45.754440 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754134    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:45.754440 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754151    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d654c48b4-zcdvc: secret "image-registry-tls" not found
Oct 30 11:29:45.754440 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754183    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:45.754440 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls podName:bebf069a-0c3e-44c6-9ad8-23dc0056c442 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.754178351 +0000 UTC m=+32.052800075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls") pod "image-registry-7d654c48b4-zcdvc" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442") : secret "image-registry-tls" not found
Oct 30 11:29:45.754440 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.754226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.754213923 +0000 UTC m=+32.052835627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:45.759260 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.759190    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-25xnv"]
Oct 30 11:29:45.803421 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.803394    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"]
Oct 30 11:29:45.809754 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:45.807379    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23360850_74e3_473e_a579_70d4fcc5a9d0.slice/crio-6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74 WatchSource:0}: Error finding container 6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74: Status 404 returned error can't find the container with id 6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74
Oct 30 11:29:45.820220 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.820196    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"]
Oct 30 11:29:45.831320 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.831300    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"]
Oct 30 11:29:45.832297 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.832277    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pnclc"]
Oct 30 11:29:45.836428 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:45.836405    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a93e48_3382_47af_adef_2ceb0d1fc6aa.slice/crio-9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df WatchSource:0}: Error finding container 9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df: Status 404 returned error can't find the container with id 9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df
Oct 30 11:29:45.836747 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:45.836726    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1d8529_d285_41d3_b179_6f3343059567.slice/crio-3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d WatchSource:0}: Error finding container 3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d: Status 404 returned error can't find the container with id 3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d
Oct 30 11:29:45.845488 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.845427    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"]
Oct 30 11:29:45.847512 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:45.847491    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5013b2f_c423_4c36_aaa0_917c35cf23f8.slice/crio-fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978 WatchSource:0}: Error finding container fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978: Status 404 returned error can't find the container with id fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978
Oct 30 11:29:45.855030 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.854991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:45.855116 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:45.855052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:45.855174 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.855142    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:45.855174 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.855165    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f7dc85469-gjnw7: secret "image-registry-tls" not found
Oct 30 11:29:45.855327 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.855215    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls podName:d58c2f33-be41-4e16-91bc-293ee27d908e nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.855194952 +0000 UTC m=+32.153816668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls") pod "image-registry-7f7dc85469-gjnw7" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e") : secret "image-registry-tls" not found
Oct 30 11:29:45.855327 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.855298    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:45.855438 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:45.855348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls podName:1d000fdc-e5ed-4fce-bb1e-6715932a37b1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:46.855332101 +0000 UTC m=+32.153953819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls") pod "dns-default-zgz8g" (UID: "1d000fdc-e5ed-4fce-bb1e-6715932a37b1") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:46.171422 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.171380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:46.186871 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.186842    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6tzdw\""
Oct 30 11:29:46.453302 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.453262    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" event={"ID":"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18","Type":"ContainerStarted","Data":"38a165ac547d5c981690b088c5b5d0c4f262adb5cd1212f0120aabe216164fef"}
Oct 30 11:29:46.459694 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.459658    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x2vn2" event={"ID":"ea88fbd7-988d-44a5-a12e-fd5e64e8da79","Type":"ContainerStarted","Data":"1fc7ea5841bff2be0b5a91f5be8a8e9bff11fa223a4c843a1bf9d3947e474846"}
Oct 30 11:29:46.459843 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.459724    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x2vn2" event={"ID":"ea88fbd7-988d-44a5-a12e-fd5e64e8da79","Type":"ContainerStarted","Data":"1c4adbf03d7af5dfe077b96ac20b5b4b81f1d0b1f1ead66fdccd24507ae607b0"}
Oct 30 11:29:46.462277 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.462250    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k" event={"ID":"5d1d8529-d285-41d3-b179-6f3343059567","Type":"ContainerStarted","Data":"3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d"}
Oct 30 11:29:46.464207 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.464172    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-25xnv" event={"ID":"cec9279a-2cd4-49bd-823c-398f6528bbe7","Type":"ContainerStarted","Data":"977868089cb1b4a8af5b255b917cfb8ae68cacfcff34a7e79a07bd1023f42c25"}
Oct 30 11:29:46.467913 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.467892    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-488kk" event={"ID":"8c68e591-1f33-4482-9843-d0412cec6bce","Type":"ContainerStarted","Data":"61d635f5321b76fc89dded535db00f3339dbe2a859a884c3a3ae17074fc7766d"}
Oct 30 11:29:46.468014 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.467927    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-488kk" event={"ID":"8c68e591-1f33-4482-9843-d0412cec6bce","Type":"ContainerStarted","Data":"f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6"}
Oct 30 11:29:46.471230 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.470760    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5b47m" event={"ID":"e3746ecb-830d-4d86-8400-21f4d2985b22","Type":"ContainerStarted","Data":"360b8df1674962a8e2923e81685d8d90653d7cb0bf3680767eec94de9ed9ce7b"}
Oct 30 11:29:46.471230 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.470802    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5b47m" event={"ID":"e3746ecb-830d-4d86-8400-21f4d2985b22","Type":"ContainerStarted","Data":"09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d"}
Oct 30 11:29:46.473111 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.473043    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq" event={"ID":"c5013b2f-c423-4c36-aaa0-917c35cf23f8","Type":"ContainerStarted","Data":"fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978"}
Oct 30 11:29:46.474681 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.474582    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pnclc" event={"ID":"22a93e48-3382-47af-adef-2ceb0d1fc6aa","Type":"ContainerStarted","Data":"9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df"}
Oct 30 11:29:46.476004 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.475978    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7" event={"ID":"52a6b071-d9c7-4d71-aeae-65e818f74dec","Type":"ContainerStarted","Data":"8071afea22a0b00e0a05bdd51ebca64a5785321a63278cf8698190acff56639e"}
Oct 30 11:29:46.477655 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.477613    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz" event={"ID":"23360850-74e3-473e-a579-70d4fcc5a9d0","Type":"ContainerStarted","Data":"6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74"}
Oct 30 11:29:46.548477 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.547813    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-x2vn2" podStartSLOduration=820.547793872 podStartE2EDuration="13m40.547793872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:46.505282859 +0000 UTC m=+31.803904581" watchObservedRunningTime="2026-10-30 11:29:46.547793872 +0000 UTC m=+31.846415597"
Oct 30 11:29:46.548477 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.548258    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-488kk" podStartSLOduration=820.548252423 podStartE2EDuration="13m40.548252423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:46.546322487 +0000 UTC m=+31.844944212" watchObservedRunningTime="2026-10-30 11:29:46.548252423 +0000 UTC m=+31.846874149"
Oct 30 11:29:46.589828 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.589770    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5b47m" podStartSLOduration=820.589748095 podStartE2EDuration="13m40.589748095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:46.589471595 +0000 UTC m=+31.888093313" watchObservedRunningTime="2026-10-30 11:29:46.589748095 +0000 UTC m=+31.888369823"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.765024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.765091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.765138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.765243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.765266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.765295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.765464    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.765529    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.765510811 +0000 UTC m=+34.064132519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.765905    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.765920    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d654c48b4-zcdvc: secret "image-registry-tls" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.765961    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls podName:bebf069a-0c3e-44c6-9ad8-23dc0056c442 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.765946945 +0000 UTC m=+34.064568652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls") pod "image-registry-7d654c48b4-zcdvc" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442") : secret "image-registry-tls" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766020    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766052    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls podName:1f5f4127-5c51-4029-8ba2-0f5c01bad573 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.766043913 +0000 UTC m=+34.064665631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lwp9b" (UID: "1f5f4127-5c51-4029-8ba2-0f5c01bad573") : secret "samples-operator-tls" not found
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766098    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.76609024 +0000 UTC m=+34.064711956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : configmap references non-existent config key: service-ca.crt
Oct 30 11:29:46.766268 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766134    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:46.767115 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766152    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.766146761 +0000 UTC m=+34.064768463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:46.767115 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766183    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:46.767115 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.766199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.766194568 +0000 UTC m=+34.064816283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:46.866948 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.866914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:46.866948 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:46.866967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:46.867146 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.867094    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:46.867146 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.867139    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls podName:1d000fdc-e5ed-4fce-bb1e-6715932a37b1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.867126011 +0000 UTC m=+34.165747713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls") pod "dns-default-zgz8g" (UID: "1d000fdc-e5ed-4fce-bb1e-6715932a37b1") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:46.867216 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.867186    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:46.867216 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.867194    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f7dc85469-gjnw7: secret "image-registry-tls" not found
Oct 30 11:29:46.867216 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:46.867214    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls podName:d58c2f33-be41-4e16-91bc-293ee27d908e nodeName:}" failed. No retries permitted until 2026-10-30 11:29:48.867207734 +0000 UTC m=+34.165829436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls") pod "image-registry-7f7dc85469-gjnw7" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e") : secret "image-registry-tls" not found
Oct 30 11:29:48.489660 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.488998    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k" event={"ID":"5d1d8529-d285-41d3-b179-6f3343059567","Type":"ContainerStarted","Data":"67527f265358702211efc73c237c582ebccbac9b324ac04edbc55880c6741a94"}
Oct 30 11:29:48.535342 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.535263    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k" podStartSLOduration=790.564538595 podStartE2EDuration="13m12.535242076s" totalImagesPullingTime="1.970703481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:48.53484187 +0000 UTC m=+33.833463595" watchObservedRunningTime="2026-10-30 11:29:48.535242076 +0000 UTC m=+33.833863799"
Oct 30 11:29:48.680251 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.680207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:48.791669 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.791592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:48.791669 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.791664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:48.791886 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.791707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:48.791886 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.791758    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:48.791886 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.791776    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d654c48b4-zcdvc: secret "image-registry-tls" not found
Oct 30 11:29:48.791886 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.791835    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls podName:bebf069a-0c3e-44c6-9ad8-23dc0056c442 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.791814953 +0000 UTC m=+38.090436675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls") pod "image-registry-7d654c48b4-zcdvc" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442") : secret "image-registry-tls" not found
Oct 30 11:29:48.791886 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.791860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.791898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.791909    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.791893734 +0000 UTC m=+38.090515436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : configmap references non-existent config key: service-ca.crt
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.791965    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.791997    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.792000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.791988796 +0000 UTC m=+38.090610583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.792045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.792064    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.792091    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls podName:1f5f4127-5c51-4029-8ba2-0f5c01bad573 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.792083141 +0000 UTC m=+38.090704843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lwp9b" (UID: "1f5f4127-5c51-4029-8ba2-0f5c01bad573") : secret "samples-operator-tls" not found
Oct 30 11:29:48.792142 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.792108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.792099194 +0000 UTC m=+38.090720895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:48.792647 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.792160    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:48.792647 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.792193    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.792181554 +0000 UTC m=+38.090803258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:48.819868 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.819748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:48.893797 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.893761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:48.894021 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.893988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:48.894277 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.894129    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:48.894277 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.894144    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f7dc85469-gjnw7: secret "image-registry-tls" not found
Oct 30 11:29:48.894277 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.894199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls podName:d58c2f33-be41-4e16-91bc-293ee27d908e nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.894181884 +0000 UTC m=+38.192803602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls") pod "image-registry-7f7dc85469-gjnw7" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e") : secret "image-registry-tls" not found
Oct 30 11:29:48.894277 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.894267    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:48.894514 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.894301    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls podName:1d000fdc-e5ed-4fce-bb1e-6715932a37b1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.894291419 +0000 UTC m=+38.192913121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls") pod "dns-default-zgz8g" (UID: "1d000fdc-e5ed-4fce-bb1e-6715932a37b1") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:48.996331 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.995206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:48.996331 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:48.995289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:48.996331 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.996011    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 11:29:48.996331 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:48.996074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs podName:e40c5726-14a1-4f26-8d5f-218b4ff946a8 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:20.996055134 +0000 UTC m=+66.294676835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs") pod "network-metrics-daemon-6s7j4" (UID: "e40c5726-14a1-4f26-8d5f-218b4ff946a8") : secret "metrics-daemon-secret" not found
Oct 30 11:29:49.000237 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.000187    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:49.088380 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.088145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:49.096912 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.096876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:49.107689 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.107631    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:49.214071 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.213583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:49.449097 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.449003    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r4xxg"]
Oct 30 11:29:49.451820 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:49.451790    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63257d8b_a976_4db1_8721_98f7d7fefed8.slice/crio-8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a WatchSource:0}: Error finding container 8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a: Status 404 returned error can't find the container with id 8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a
Oct 30 11:29:49.496221 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.496152    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r4xxg" event={"ID":"63257d8b-a976-4db1-8721-98f7d7fefed8","Type":"ContainerStarted","Data":"8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a"}
Oct 30 11:29:49.555904 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:49.555772    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxtn4"]
Oct 30 11:29:49.678213 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:49.678163    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8a23a0_a643_42bd_84f7_29a2bb9138b7.slice/crio-f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce WatchSource:0}: Error finding container f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce: Status 404 returned error can't find the container with id f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce
Oct 30 11:29:50.501471 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.500591    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-25xnv" event={"ID":"cec9279a-2cd4-49bd-823c-398f6528bbe7","Type":"ContainerStarted","Data":"b0c4044544a5156369b74978b964fe1251af2e030f7a118a58f537d40701cbb6"}
Oct 30 11:29:50.513785 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.503230    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxtn4" event={"ID":"cf8a23a0-a643-42bd-84f7-29a2bb9138b7","Type":"ContainerStarted","Data":"f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce"}
Oct 30 11:29:50.513785 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.505473    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz" event={"ID":"23360850-74e3-473e-a579-70d4fcc5a9d0","Type":"ContainerStarted","Data":"3f249043c4e3ab6c8f4668aed03cf189b0495fe6309f2e2177cd02546f1e20be"}
Oct 30 11:29:50.551402 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.551327    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-25xnv" podStartSLOduration=790.604580951 podStartE2EDuration="13m14.551311037s" totalImagesPullingTime="3.946730086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:50.549668578 +0000 UTC m=+35.848290307" watchObservedRunningTime="2026-10-30 11:29:50.551311037 +0000 UTC m=+35.849932766"
Oct 30 11:29:50.592149 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.592089    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz" podStartSLOduration=790.367777327 podStartE2EDuration="13m14.592074771s" totalImagesPullingTime="4.224297444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:50.590927415 +0000 UTC m=+35.889549162" watchObservedRunningTime="2026-10-30 11:29:50.592074771 +0000 UTC m=+35.890696499"
Oct 30 11:29:50.741792 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.741703    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x9lhg"]
Oct 30 11:29:50.756616 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.756595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:50.765176 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.765131    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x9lhg"]
Oct 30 11:29:50.774659 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.774438    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 11:29:50.774659 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.774590    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkn76\""
Oct 30 11:29:50.774659 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.774642    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 11:29:50.816272 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.816242    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t6jts-b62vt"]
Oct 30 11:29:50.838792 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.838656    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t6jts-b62vt"]
Oct 30 11:29:50.838792 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.838741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:50.859506 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.859476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:50.874649 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.874622    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lfmd8\""
Oct 30 11:29:50.921951 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.921920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5qs\" (UniqueName: \"kubernetes.io/projected/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-api-access-mr5qs\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:50.922075 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.922018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da62aabf-4741-4ada-b763-f5d62d142ae4-crio-socket\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:50.922134 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.922106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:50.922190 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.922163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da62aabf-4741-4ada-b763-f5d62d142ae4-data-volume\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:50.922240 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:50.922196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.032803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da62aabf-4741-4ada-b763-f5d62d142ae4-data-volume\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.032853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.032914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.032969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.032997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5qs\" (UniqueName: \"kubernetes.io/projected/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-api-access-mr5qs\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.033055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.033093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da62aabf-4741-4ada-b763-f5d62d142ae4-crio-socket\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.033114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.033166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.033278    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.033331    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:51.533313681 +0000 UTC m=+36.831935403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.033863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da62aabf-4741-4ada-b763-f5d62d142ae4-data-volume\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.033942    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.033987    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:51.533972802 +0000 UTC m=+36.832594518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:51.034555 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.034304    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da62aabf-4741-4ada-b763-f5d62d142ae4-crio-socket\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.050354 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.050280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:51.063267 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.063240    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5qs\" (UniqueName: \"kubernetes.io/projected/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-api-access-mr5qs\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.111714 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.111680    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"]
Oct 30 11:29:51.130308 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.130277    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"]
Oct 30 11:29:51.131314 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.131298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.134152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.134219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.134377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.134430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.135408    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.135508    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.136065 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.136010    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.147871 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.147750    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 11:29:51.147871 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.147830    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k8lzt\""
Oct 30 11:29:51.147871 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.147837    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:51.151004 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.150584    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"]
Oct 30 11:29:51.151004 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.150620    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"]
Oct 30 11:29:51.151004 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.150717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:51.167392 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.167366    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.174090 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.174070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:51.226027 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.225932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:51.235893 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.235862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbfd\" (UniqueName: \"kubernetes.io/projected/b070df60-6cdb-43fa-842b-ca11d66b1770-kube-api-access-hpbfd\") pod \"migrator-66df5d496b-brq9g\" (UID: \"b070df60-6cdb-43fa-842b-ca11d66b1770\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:51.336780 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.336704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dh4\" (UniqueName: \"kubernetes.io/projected/a63e5bbb-4666-48a9-8530-64848d5bf70b-kube-api-access-q7dh4\") pod \"migrator-66df5d496b-v4nfq\" (UID: \"a63e5bbb-4666-48a9-8530-64848d5bf70b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:51.336943 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.336813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbfd\" (UniqueName: \"kubernetes.io/projected/b070df60-6cdb-43fa-842b-ca11d66b1770-kube-api-access-hpbfd\") pod \"migrator-66df5d496b-brq9g\" (UID: \"b070df60-6cdb-43fa-842b-ca11d66b1770\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:51.361298 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.361267    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbfd\" (UniqueName: \"kubernetes.io/projected/b070df60-6cdb-43fa-842b-ca11d66b1770-kube-api-access-hpbfd\") pod \"migrator-66df5d496b-brq9g\" (UID: \"b070df60-6cdb-43fa-842b-ca11d66b1770\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:51.386439 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.386394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:51.439518 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.438475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dh4\" (UniqueName: \"kubernetes.io/projected/a63e5bbb-4666-48a9-8530-64848d5bf70b-kube-api-access-q7dh4\") pod \"migrator-66df5d496b-v4nfq\" (UID: \"a63e5bbb-4666-48a9-8530-64848d5bf70b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:51.461761 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.461729    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dh4\" (UniqueName: \"kubernetes.io/projected/a63e5bbb-4666-48a9-8530-64848d5bf70b-kube-api-access-q7dh4\") pod \"migrator-66df5d496b-v4nfq\" (UID: \"a63e5bbb-4666-48a9-8530-64848d5bf70b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:51.463589 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.463564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:51.482743 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.482718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:51.513372 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.512339    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq" event={"ID":"c5013b2f-c423-4c36-aaa0-917c35cf23f8","Type":"ContainerStarted","Data":"0fa59d675acc9019d4772fc45f0b3d2b0e146dab7f0b1b3cb78e8e0e2bb2f86d"}
Oct 30 11:29:51.515124 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.515077    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/0.log"
Oct 30 11:29:51.515230 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.515126    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" containerID="241975490af818ed85eb674272d2e5a091f47ad786deee1fbb640f75c22c53d4" exitCode=255
Oct 30 11:29:51.515536 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.515514    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" event={"ID":"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18","Type":"ContainerDied","Data":"241975490af818ed85eb674272d2e5a091f47ad786deee1fbb640f75c22c53d4"}
Oct 30 11:29:51.515920 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.515903    2579 scope.go:122] "RemoveContainer" containerID="241975490af818ed85eb674272d2e5a091f47ad786deee1fbb640f75c22c53d4"
Oct 30 11:29:51.520122 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.520107    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t6jts-b62vt"]
Oct 30 11:29:51.521836 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:51.521813    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c980147_d63b_436d_bff6_5a458249fe5a.slice/crio-38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee WatchSource:0}: Error finding container 38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee: Status 404 returned error can't find the container with id 38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee
Oct 30 11:29:51.539304 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.539274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.540733 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.539479    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:51.540733 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.539541    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.539523046 +0000 UTC m=+37.838144763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:51.540733 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.539855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:51.540733 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.539969    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:51.540733 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:51.540017    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.540002782 +0000 UTC m=+37.838624485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:51.545426 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.545402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:51.557640 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.557596    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq" podStartSLOduration=790.717188157 podStartE2EDuration="13m15.557578711s" totalImagesPullingTime="4.840390554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:51.555788051 +0000 UTC m=+36.854409776" watchObservedRunningTime="2026-10-30 11:29:51.557578711 +0000 UTC m=+36.856200435"
Oct 30 11:29:51.682276 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.682087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:51.849155 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.849117    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"]
Oct 30 11:29:51.852617 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.852594    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"]
Oct 30 11:29:51.875382 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:51.875318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:51.922316 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:51.922278    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63e5bbb_4666_48a9_8530_64848d5bf70b.slice/crio-c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5 WatchSource:0}: Error finding container c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5: Status 404 returned error can't find the container with id c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5
Oct 30 11:29:51.923491 ip-10-0-142-105 kubenswrapper[2579]: W1030 11:29:51.923442    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb070df60_6cdb_43fa_842b_ca11d66b1770.slice/crio-f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc WatchSource:0}: Error finding container f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc: Status 404 returned error can't find the container with id f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc
Oct 30 11:29:52.096895 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.096815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:52.524470 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.524416    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6jts-b62vt" event={"ID":"3c980147-d63b-436d-bff6-5a458249fe5a","Type":"ContainerStarted","Data":"6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3"}
Oct 30 11:29:52.524977 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.524500    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6jts-b62vt" event={"ID":"3c980147-d63b-436d-bff6-5a458249fe5a","Type":"ContainerStarted","Data":"38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee"}
Oct 30 11:29:52.527831 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.527409    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pnclc" event={"ID":"22a93e48-3382-47af-adef-2ceb0d1fc6aa","Type":"ContainerStarted","Data":"941707c861a6ce7a0625ecb203750134dbfb0e3c23fd8d9f397fba92f78b9d67"}
Oct 30 11:29:52.529081 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.529016    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g" event={"ID":"b070df60-6cdb-43fa-842b-ca11d66b1770","Type":"ContainerStarted","Data":"f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc"}
Oct 30 11:29:52.531076 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.530680    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7" event={"ID":"52a6b071-d9c7-4d71-aeae-65e818f74dec","Type":"ContainerStarted","Data":"9a65c297bdda28c899cc1f28ca934483596a294342e48d4df88f6ae0931ca0b1"}
Oct 30 11:29:52.536149 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.536071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/1.log"
Oct 30 11:29:52.537690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.536746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/0.log"
Oct 30 11:29:52.537690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.536779    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572" exitCode=255
Oct 30 11:29:52.537690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.536871    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" event={"ID":"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18","Type":"ContainerDied","Data":"c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572"}
Oct 30 11:29:52.537690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.536923    2579 scope.go:122] "RemoveContainer" containerID="241975490af818ed85eb674272d2e5a091f47ad786deee1fbb640f75c22c53d4"
Oct 30 11:29:52.537690 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.537251    2579 scope.go:122] "RemoveContainer" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572"
Oct 30 11:29:52.537690 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.537428    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scqxd_openshift-console-operator(c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18"
Oct 30 11:29:52.539918 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.539863    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq" event={"ID":"a63e5bbb-4666-48a9-8530-64848d5bf70b","Type":"ContainerStarted","Data":"c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5"}
Oct 30 11:29:52.556111 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.556059    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:52.556214 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.556171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:54.556150675 +0000 UTC m=+39.854772394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:52.556353 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.556331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:52.556470 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.556433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:52.557250 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.557229    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:52.557331 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.557285    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:54.557268942 +0000 UTC m=+39.855890658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:52.655334 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.655269    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t6jts-b62vt" podStartSLOduration=2.655248414 podStartE2EDuration="2.655248414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:52.60259509 +0000 UTC m=+37.901216813" watchObservedRunningTime="2026-10-30 11:29:52.655248414 +0000 UTC m=+37.953870140"
Oct 30 11:29:52.701562 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.701493    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pnclc" podStartSLOduration=1.996141996 podStartE2EDuration="7.701473194s" totalImagesPullingTime="5.705331198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:52.700198889 +0000 UTC m=+37.998820613" watchObservedRunningTime="2026-10-30 11:29:52.701473194 +0000 UTC m=+38.000094918"
Oct 30 11:29:52.701799 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.701762    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7" podStartSLOduration=788.566322662 podStartE2EDuration="13m14.701755681s" totalImagesPullingTime="6.135433019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:52.656844137 +0000 UTC m=+37.955465860" watchObservedRunningTime="2026-10-30 11:29:52.701755681 +0000 UTC m=+38.000377410"
Oct 30 11:29:52.763378 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.763346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:52.859528 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.859439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:52.859528 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.859495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:52.859765 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.859534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:52.859765 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.859561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:52.859765 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.859594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:52.859765 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.859621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:52.860011 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.859792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.859779209 +0000 UTC m=+46.158400911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : configmap references non-existent config key: service-ca.crt
Oct 30 11:29:52.860162 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860120    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:52.860215 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860165    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.860151158 +0000 UTC m=+46.158772863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:52.860215 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860209    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:52.860313 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860237    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.86023013 +0000 UTC m=+46.158851854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:52.860371 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860318    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 11:29:52.860371 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860341    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls podName:1f5f4127-5c51-4029-8ba2-0f5c01bad573 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.860335102 +0000 UTC m=+46.158956804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lwp9b" (UID: "1f5f4127-5c51-4029-8ba2-0f5c01bad573") : secret "samples-operator-tls" not found
Oct 30 11:29:52.860497 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860379    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:52.860497 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860384    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:52.860497 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860406    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d654c48b4-zcdvc: secret "image-registry-tls" not found
Oct 30 11:29:52.860497 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860397    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.860392245 +0000 UTC m=+46.159013948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:52.860497 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.860484    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls podName:bebf069a-0c3e-44c6-9ad8-23dc0056c442 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.860475209 +0000 UTC m=+46.159096911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls") pod "image-registry-7d654c48b4-zcdvc" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442") : secret "image-registry-tls" not found
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.960674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.960884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.961007    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.961021    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f7dc85469-gjnw7: secret "image-registry-tls" not found
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.961076    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls podName:d58c2f33-be41-4e16-91bc-293ee27d908e nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.96105906 +0000 UTC m=+46.259680766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls") pod "image-registry-7f7dc85469-gjnw7" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e") : secret "image-registry-tls" not found
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.961140    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:52.961243 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:52.961169    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls podName:1d000fdc-e5ed-4fce-bb1e-6715932a37b1 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:00.961159971 +0000 UTC m=+46.259781677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls") pod "dns-default-zgz8g" (UID: "1d000fdc-e5ed-4fce-bb1e-6715932a37b1") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:52.999731 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:52.999644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:53.230283 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:53.230242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:53.548040 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:53.547968    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/1.log"
Oct 30 11:29:53.548523 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:53.548367    2579 scope.go:122] "RemoveContainer" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572"
Oct 30 11:29:53.548652 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:53.548630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scqxd_openshift-console-operator(c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18"
Oct 30 11:29:53.859101 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:53.859023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:54.553444 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.553400    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g" event={"ID":"b070df60-6cdb-43fa-842b-ca11d66b1770","Type":"ContainerStarted","Data":"29984bfff5d3bb2b676bba981b9aea4b10c6841e2dd51b0a653cb380467eff26"}
Oct 30 11:29:54.555289 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.555254    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq" event={"ID":"a63e5bbb-4666-48a9-8530-64848d5bf70b","Type":"ContainerStarted","Data":"25e53b3ceab6df8f7df33b4961852139736858fc6b0767ec9a453010858b27f2"}
Oct 30 11:29:54.556916 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.556887    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxtn4" event={"ID":"cf8a23a0-a643-42bd-84f7-29a2bb9138b7","Type":"ContainerStarted","Data":"a5eb1823dbfa10f1dbc6bbe7e75cd1098d1e5466ce9de47058da95506b091e79"}
Oct 30 11:29:54.557053 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.557040    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:54.578628 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.578599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:54.578732 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.578699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:54.578732 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:54.578717    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:54.578824 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:54.578807    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.578789945 +0000 UTC m=+43.877411647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:54.578881 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:54.578852    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:54.578929 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:54.578904    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.578889233 +0000 UTC m=+43.877510939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:54.598280 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.598239    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g" podStartSLOduration=1.5271884949999999 podStartE2EDuration="3.598227304s" totalImagesPullingTime="2.071038809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:54.596038701 +0000 UTC m=+39.894660425" watchObservedRunningTime="2026-10-30 11:29:54.598227304 +0000 UTC m=+39.896849029"
Oct 30 11:29:54.637662 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.637614    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq" podStartSLOduration=1.569232884 podStartE2EDuration="3.637594066s" totalImagesPullingTime="2.068361182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:54.636600633 +0000 UTC m=+39.935222358" watchObservedRunningTime="2026-10-30 11:29:54.637594066 +0000 UTC m=+39.936215791"
Oct 30 11:29:54.678334 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:54.678288    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vxtn4" podStartSLOduration=34.740684988 podStartE2EDuration="38.67827406s" totalImagesPullingTime="3.937589072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:54.677246614 +0000 UTC m=+39.975868340" watchObservedRunningTime="2026-10-30 11:29:54.67827406 +0000 UTC m=+39.976895787"
Oct 30 11:29:55.375003 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:55.374964    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:55.375175 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:55.375029    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:55.375385 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:55.375366    2579 scope.go:122] "RemoveContainer" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572"
Oct 30 11:29:55.375619 ip-10-0-142-105 kubenswrapper[2579]: E1030 11:29:55.375600    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scqxd_openshift-console-operator(c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18"
Oct 30 11:29:56.092344 ip-10-0-142-105 kubenswrapper[2579]: I1030 11:29:56.092276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lsq8_5815b6d3-0bdd-45b2-a673-5f493146c0f8/dns-node-resolver/0.log"
Oct 30 11:29:56.147194 ip-10-0-142-105 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 11:29:56.162863 ip-10-0-142-105 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 11:29:56.163033 ip-10-0-142-105 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 11:29:56.163085 ip-10-0-142-105 systemd[1]: kubelet.service: Consumed 3.153s CPU time.
Oct 30 11:29:56.173914 ip-10-0-142-105 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 11:29:56.246183 ip-10-0-142-105 kubenswrapper[5060]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:56.246183 ip-10-0-142-105 kubenswrapper[5060]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 11:29:56.246183 ip-10-0-142-105 kubenswrapper[5060]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:56.246183 ip-10-0-142-105 kubenswrapper[5060]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:56.248422 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248331    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.248422 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248418    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.248422 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248423    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.248422 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248428    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248431    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248435    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248439    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248445    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248470    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248474    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248478    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248482    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248485    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248489    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248493    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248496    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248500    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248503    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248507    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248511    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248515    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248519    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.248731 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248523    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248527    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248530    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248542    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248546    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248550    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248555    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248559    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248563    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248567    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248570    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248575    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248578    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248583    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248587    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248593    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248601    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248607    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248612    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.249566 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248616    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248620    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248625    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248629    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248634    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248638    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248642    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248646    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248655    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248661    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248665    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248670    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248674    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248679    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248683    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248687    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248691    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248695    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248699    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.250408 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248703    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248708    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248712    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248717    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248721    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248725    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248729    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248733    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248737    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248741    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248745    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248750    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248755    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248759    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248764    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248768    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248772    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248776    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248780    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248784    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.251285 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248788    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248793    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248797    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248801    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248805    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248809    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248813    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248817    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248821    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248826    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248830    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248835    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248839    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248843    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248847    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248852    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248856    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248860    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248864    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.252182 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248868    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248873    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248877    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248883    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248888    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248892    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.248897    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250258    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250268    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250273    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250279    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250284    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250288    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250293    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250297    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250302    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250307    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250311    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250315    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250319    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.252732 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250323    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250328    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250332    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250337    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250341    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250346    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250350    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250354    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250358    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250362    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250366    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250371    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250375    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250379    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250384    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250388    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250393    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250397    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250402    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.253296 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250407    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250411    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250415    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250419    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250423    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250429    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250434    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250439    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250443    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250467    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250472    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250477    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250481    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250485    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250489    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250495    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250499    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250503    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250508    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250512    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.253798 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250516    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250521    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250525    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250529    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250536    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250543    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250549    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250554    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250559    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250563    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250567    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250572    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250576    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250581    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250595    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250599    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250604    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250609    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250613    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.254324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250617    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250621    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250625    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250630    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250634    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250638    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250642    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250646    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250651    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250655    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250660    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250664    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250668    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250672    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250677    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250681    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250685    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250689    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250693    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250698    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.254808 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250702    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250706    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250710    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250714    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250718    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250722    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250726    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250731    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250735    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250739    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250745    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250750    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250754    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250758    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.250762    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251587    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251596    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251601    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251606    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251610    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.255315 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251614    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251619    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251623    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251628    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251632    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251636    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251640    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251644    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251649    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251653    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251657    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251661    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251665    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251669    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251673    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251677    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251685    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251690    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251694    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.255922 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251699    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251705    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251712    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251716    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251721    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251726    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251731    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251736    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251741    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251747    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251751    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251755    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251759    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251763    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251768    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251772    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251777    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251782    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251786    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.256381 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251790    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251794    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251798    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251803    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251806    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251811    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251815    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251819    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251824    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251828    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251835    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251839    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251843    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251847    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251852    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251856    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251861    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251865    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251869    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251873    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.256919 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251877    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251881    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251885    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251890    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251894    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251898    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251902    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251907    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251911    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251915    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251919    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251923    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251927    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251931    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251935    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251939    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251944    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251948    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251952    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251956    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.257702 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251960    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251966    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251973    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251978    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251982    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251988    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251992    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.251997    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252001    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252005    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252010    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252014    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252018    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252022    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252026    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252030    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252035    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252039    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252043    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.258355 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252047    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252051    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252059    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252063    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252202    5060 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252231    5060 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252239    5060 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252243    5060 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252248    5060 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252251    5060 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252256    5060 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252261    5060 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252264    5060 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252267    5060 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252270    5060 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252273    5060 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252276    5060 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252279    5060 flags.go:64] FLAG: --cgroup-root=""
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252284    5060 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252288    5060 flags.go:64] FLAG: --client-ca-file=""
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252293    5060 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252296    5060 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252300    5060 flags.go:64] FLAG: --cluster-domain=""
Oct 30 11:29:56.258873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252303    5060 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252306    5060 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252310    5060 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252313    5060 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252318    5060 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252321    5060 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252324    5060 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252327    5060 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252330    5060 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252334    5060 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252337    5060 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252340    5060 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252343    5060 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252347    5060 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252352    5060 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252354    5060 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252357    5060 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252360    5060 flags.go:64] FLAG: --enable-server="true"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252363    5060 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252368    5060 flags.go:64] FLAG: --event-burst="100"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252371    5060 flags.go:64] FLAG: --event-qps="50"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252375    5060 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252378    5060 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252381    5060 flags.go:64] FLAG: --eviction-hard=""
Oct 30 11:29:56.259463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252385    5060 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252388    5060 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252391    5060 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252394    5060 flags.go:64] FLAG: --eviction-soft=""
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252397    5060 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252400    5060 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252403    5060 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252406    5060 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252409    5060 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252412    5060 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252415    5060 flags.go:64] FLAG: --feature-gates=""
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252419    5060 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252422    5060 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252425    5060 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252428    5060 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252431    5060 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252434    5060 flags.go:64] FLAG: --help="false"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252438    5060 flags.go:64] FLAG: --hostname-override="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252441    5060 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252444    5060 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252447    5060 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252469    5060 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252473    5060 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252476    5060 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 11:29:56.260087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252479    5060 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252482    5060 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252486    5060 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252489    5060 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252492    5060 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252495    5060 flags.go:64] FLAG: --kube-reserved=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252498    5060 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252501    5060 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252504    5060 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252506    5060 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252509    5060 flags.go:64] FLAG: --lock-file=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252512    5060 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252515    5060 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252518    5060 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252523    5060 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252526    5060 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252529    5060 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252532    5060 flags.go:64] FLAG: --logging-format="text"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252544    5060 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252548    5060 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252551    5060 flags.go:64] FLAG: --manifest-url=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252557    5060 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252561    5060 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252564    5060 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252568    5060 flags.go:64] FLAG: --max-pods="110"
Oct 30 11:29:56.260717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252571    5060 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252574    5060 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252577    5060 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252580    5060 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252583    5060 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252586    5060 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252589    5060 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252598    5060 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252601    5060 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252604    5060 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252607    5060 flags.go:64] FLAG: --pod-cidr=""
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252610    5060 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252613    5060 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252616    5060 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252618    5060 flags.go:64] FLAG: --port="10250"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252621    5060 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252624    5060 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a72f43620cbd57b8"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252628    5060 flags.go:64] FLAG: --qos-reserved=""
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252630    5060 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252634    5060 flags.go:64] FLAG: --register-node="true"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252636    5060 flags.go:64] FLAG: --register-with-taints=""
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252640    5060 flags.go:64] FLAG: --registry-burst="10"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252643    5060 flags.go:64] FLAG: --registry-qps="5"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252646    5060 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252648    5060 flags.go:64] FLAG: --reserved-memory=""
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252656    5060 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 11:29:56.261367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252659    5060 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252662    5060 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252666    5060 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252669    5060 flags.go:64] FLAG: --runonce="false"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252672    5060 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252675    5060 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252678    5060 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252681    5060 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252683    5060 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252686    5060 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252689    5060 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252692    5060 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252695    5060 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252698    5060 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252701    5060 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252704    5060 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252707    5060 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252710    5060 flags.go:64] FLAG: --system-cgroups=""
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252712    5060 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252718    5060 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252721    5060 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252724    5060 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252727    5060 flags.go:64] FLAG: --tls-min-version=""
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252730    5060 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252733    5060 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 11:29:56.262085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252736    5060 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252739    5060 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252742    5060 flags.go:64] FLAG: --v="2"
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252746    5060 flags.go:64] FLAG: --version="false"
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252750    5060 flags.go:64] FLAG: --vmodule=""
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252754    5060 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.252758    5060 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252900    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252903    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252908    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252911    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252913    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252916    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252918    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252921    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252923    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252926    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252928    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252931    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252933    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252936    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.262801 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252938    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252942    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252945    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252947    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252952    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252955    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252958    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252961    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252963    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252965    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252968    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252970    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252973    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252975    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252978    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.252980    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253000    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253007    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253012    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253016    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.263569 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253025    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253029    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253034    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253037    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253041    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253045    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253049    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253052    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253057    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253061    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253064    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253068    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253072    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253075    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253080    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253084    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253088    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253092    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253096    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.264231 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253100    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253104    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253108    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253111    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253115    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253119    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253123    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253127    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253131    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253136    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253141    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253145    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253149    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253154    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253158    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253162    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253166    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253169    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253174    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.264857 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253177    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253181    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253185    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253189    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253193    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253196    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253200    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253204    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253208    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253213    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253216    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253220    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253224    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253229    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253234    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253238    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253241    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253243    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253246    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253249    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.265321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253251    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253254    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253256    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253260    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253263    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253266    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253269    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253272    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253275    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253277    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253280    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253282    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253285    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253287    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.253322    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:56.265939 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253362    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253366    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253370    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253374    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253377    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253380    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253383    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253386    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253388    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253391    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253393    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253396    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253398    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253401    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253403    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253406    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253408    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253410    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253413    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.266443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253415    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253418    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253422    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253424    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253428    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253431    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253433    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253436    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253438    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253441    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253443    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253445    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253473    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253477    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253480    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253482    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253485    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253488    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253490    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.266974 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253493    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253495    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253499    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253502    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253504    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253506    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253509    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253511    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253514    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253516    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253519    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253521    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253523    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253526    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253529    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253532    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253535    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253539    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253541    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.267506 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253544    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253546    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253549    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253551    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253554    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253556    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253559    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253561    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253564    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253566    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253569    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253572    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253574    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253579    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253582    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253585    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253588    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253590    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253593    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253595    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.267975 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253598    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253600    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253603    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253605    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253607    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253610    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253612    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253615    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253618    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253621    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253624    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253627    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253629    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253632    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253634    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253637    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253640    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253642    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253644    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.268492 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253647    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253649    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253652    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253654    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253657    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253659    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253661    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253664    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253666    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.253670    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.258718    5060 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.258736    5060 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.258754    5060 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.258759    5060 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258817    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258823    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258826    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258829    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258831    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258834    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.268973 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258836    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258839    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258842    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258844    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258847    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258850    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258852    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258855    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258858    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258865    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258868    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258871    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258874    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258877    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258880    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258885    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258890    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258893    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258896    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.269511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258899    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258902    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258905    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258908    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258910    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258913    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258915    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258918    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258920    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258923    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258925    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258928    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258930    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258933    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258936    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258938    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258941    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258943    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258946    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258949    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.269980 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258951    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258954    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258957    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258960    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258962    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258965    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258968    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258970    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258973    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258976    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258978    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258981    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258983    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258986    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258988    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258991    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258993    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258996    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.258999    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.270493 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259001    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259004    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259007    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259010    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259014    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259016    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259019    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259022    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259024    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259028    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259031    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259033    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259035    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259038    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259040    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259043    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259046    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259049    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259051    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259054    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.271113 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259057    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259060    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259062    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259065    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259067    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259070    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259073    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259075    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259077    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259080    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259082    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259085    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259087    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259089    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259092    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259094    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259096    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259099    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259101    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259104    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.271747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259107    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259109    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.259139    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259294    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259299    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259303    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259306    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259309    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259312    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259315    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259317    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259320    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259323    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259325    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259329    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259331    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.272253 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259333    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259336    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259339    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259341    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259344    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259346    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259349    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259351    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259354    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259356    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259358    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259361    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259363    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259366    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259368    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259370    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259373    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259376    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259378    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.272827 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259381    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259383    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259386    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259388    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259391    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259393    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259396    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259399    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259401    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259403    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259406    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259408    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259411    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259414    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259416    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259419    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259421    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259424    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259426    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259429    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.273403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259431    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259434    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259436    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259439    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259441    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259444    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259446    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259463    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259466    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259469    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259471    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259475    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259478    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259480    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259483    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259486    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259488    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259491    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259493    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.274229 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259497    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259499    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259502    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259504    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259507    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259509    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259511    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259514    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259516    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259519    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259522    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259524    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259527    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259529    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259532    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259535    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259539    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259542    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259546    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.275184 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259550    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259553    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259556    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259559    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259562    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259564    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259567    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259569    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259572    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259575    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259578    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259580    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259583    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259585    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259588    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259591    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.275747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.259614    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259647    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259651    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259654    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259656    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259659    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259661    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259664    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259666    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259669    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259672    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259674    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259677    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259679    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259682    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259684    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259686    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259689    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259691    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259694    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:56.276309 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259696    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259699    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259701    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259704    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259706    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259708    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259711    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259713    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259716    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259718    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259720    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259723    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259726    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259728    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259731    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259733    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259735    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259738    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259740    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259742    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:56.276864 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259745    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259747    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259751    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259753    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259756    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259758    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259761    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259763    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259766    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259768    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259771    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259773    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259776    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259778    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259780    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259783    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259785    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259788    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259790    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:56.277361 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259793    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259795    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259798    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259800    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259802    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259805    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259808    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259810    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259813    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259819    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259821    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259824    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259826    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259828    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259831    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259834    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259836    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259838    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259841    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259843    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:56.277892 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259846    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259848    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259851    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259854    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259858    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259862    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259864    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259868    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259871    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259874    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259877    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259879    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259881    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259884    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259886    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259889    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259891    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259894    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259897    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:56.278402 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259900    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259902    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259907    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259910    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259912    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259915    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259918    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259920    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:56.259923    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.259978    5060 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.260953    5060 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.261017    5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.261186    5060 server.go:1014] "Starting client certificate rotation"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.261335    5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.261401    5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:24:15 +0000 UTC" deadline="2028-07-04 22:48:12.78627865 +0000 UTC"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.261440    5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14723h18m16.524841756s"
Oct 30 11:29:56.278912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.261664    5060 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:29:56.279555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.262707    5060 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:29:56.279555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.264030    5060 log.go:25] "Validated CRI v1 runtime API"
Oct 30 11:29:56.279555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.265694    5060 log.go:25] "Validated CRI v1 image API"
Oct 30 11:29:56.279555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.266415    5060 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 11:29:56.279555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.268332    5060 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad0f2dce-b61c-4511-a05d-9671989f331b:/dev/nvme0n1p4 e1195ac5-62c2-4f55-9cca-55bfec6ca0b5:/dev/nvme0n1p3]
Oct 30 11:29:56.279814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.268352    5060 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1255d8a23fb23bd450dcafa6722e054f47a68521a7ac6c8e915cd6edf30c546d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1255d8a23fb23bd450dcafa6722e054f47a68521a7ac6c8e915cd6edf30c546d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c4adbf03d7af5dfe077b96ac20b5b4b81f1d0b1f1ead66fdccd24507ae607b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c4adbf03d7af5dfe077b96ac20b5b4b81f1d0b1f1ead66fdccd24507ae607b0/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38a165ac547d5c981690b088c5b5d0c4f262adb5cd1212f0120aabe216164fef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38a165ac547d5c981690b088c5b5d0c4f262adb5cd1212f0120aabe216164fef/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8071afea22a0b00e0a05bdd51ebca64a5785321a63278cf8698190acff56639e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8071afea22a0b00e0a05bdd51ebca64a5785321a63278cf8698190acff56639e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/977868089cb1b4a8af5b255b917cfb8ae68cacfcff34a7e79a07bd1023f42c25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/977868089cb1b4a8af5b255b917cfb8ae68cacfcff34a7e79a07bd1023f42c25/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f4370e6-112e-4d65-940f-5ca5263ce913/volumes/kubernetes.io~projected/kube-api-access-gg5n6:{mountpoint:/var/lib/kubelet/pods/0f4370e6-112e-4d65-940f-5ca5263ce913/volumes/kubernetes.io~projected/kube-api-access-gg5n6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~projected/kube-api-access-tbt8b:{mountpoint:/var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~projected/kube-api-access-tbt8b major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d000fdc-e5ed-4fce-bb1e-6715932a37b1/volumes/kubernetes.io~projected/kube-api-access-b59qz:{mountpoint:/var/lib/kubelet/pods/1d000fdc-e5ed-4fce-bb1e-6715932a37b1/volumes/kubernetes.io~projected/kube-api-access-b59qz major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f5f4127-5c51-4029-8ba2-0f5c01bad573/volumes/kubernetes.io~projected/kube-api-access-lvg6b:{mountpoint:/var/lib/kubelet/pods/1f5f4127-5c51-4029-8ba2-0f5c01bad573/volumes/kubernetes.io~projected/kube-api-access-lvg6b major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22a93e48-3382-47af-adef-2ceb0d1fc6aa/volumes/kubernetes.io~projected/kube-api-access-5wzgg:{mountpoint:/var/lib/kubelet/pods/22a93e48-3382-47af-adef-2ceb0d1fc6aa/volumes/kubernetes.io~projected/kube-api-access-5wzgg major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22a93e48-3382-47af-adef-2ceb0d1fc6aa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/22a93e48-3382-47af-adef-2ceb0d1fc6aa/volumes/kubernetes.io~secret/cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23360850-74e3-473e-a579-70d4fcc5a9d0/volumes/kubernetes.io~projected/kube-api-access-6gvld:{mountpoint:/var/lib/kubelet/pods/23360850-74e3-473e-a579-70d4fcc5a9d0/volumes/kubernetes.io~projected/kube-api-access-6gvld major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23360850-74e3-473e-a579-70d4fcc5a9d0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/23360850-74e3-473e-a579-70d4fcc5a9d0/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c980147-d63b-436d-bff6-5a458249fe5a/volumes/kubernetes.io~projected/kube-api-access-lb779:{mountpoint:/var/lib/kubelet/pods/3c980147-d63b-436d-bff6-5a458249fe5a/volumes/kubernetes.io~projected/kube-api-access-lb779 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46099da2-dc06-45b7-bc3c-139b84c896e0/volumes/kubernetes.io~projected/kube-api-access-v7jdn:{mountpoint:/var/lib/kubelet/pods/46099da2-dc06-45b7-bc3c-139b84c896e0/volumes/kubernetes.io~projected/kube-api-access-v7jdn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a6b071-d9c7-4d71-aeae-65e818f74dec/volumes/kubernetes.io~projected/kube-api-access-q4mfs:{mountpoint:/var/lib/kubelet/pods/52a6b071-d9c7-4d71-aeae-65e818f74dec/volumes/kubernetes.io~projected/kube-api-access-q4mfs major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5339a2af-2f4a-4444-b146-dcb849fa7449/volumes/kubernetes.io~projected/kube-api-access-hzwtx:{mountpoint:/var/lib/kubelet/pods/5339a2af-2f4a-4444-b146-dcb849fa7449/volumes/kubernetes.io~projected/kube-api-access-hzwtx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5815b6d3-0bdd-45b2-a673-5f493146c0f8/volumes/kubernetes.io~projected/kube-api-access-jbrt9:{mountpoint:/var/lib/kubelet/pods/5815b6d3-0bdd-45b2-a673-5f493146c0f8/volumes/kubernetes.io~projected/kube-api-access-jbrt9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~projected/kube-api-access-bjcb6:{mountpoint:/var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~projected/kube-api-access-bjcb6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d1d8529-d285-41d3-b179-6f3343059567/volumes/kubernetes.io~projected/kube-api-access-bmzlw:{mountpoint:/var/lib/kubelet/pods/5d1d8529-d285-41d3-b179-6f3343059567/volumes/kubernetes.io~projected/kube-api-access-bmzlw major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63257d8b-a976-4db1-8721-98f7d7fefed8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/63257d8b-a976-4db1-8721-98f7d7fefed8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c68e591-1f33-4482-9843-d0412cec6bce/volumes/kubernetes.io~projected/kube-api-access-nr5b8:{mountpoint:/var/lib/kubelet/pods/8c68e591-1f33-4482-9843-d0412cec6bce/volumes/kubernetes.io~projected/kube-api-access-nr5b8 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91cfe548-7bc2-4679-99a3-7212bdc92193/volumes/kubernetes.io~projected/kube-api-access-zdhn7:{mountpoint:/var/lib/kubelet/pods/91cfe548-7bc2-4679-99a3-7212bdc92193/volumes/kubernetes.io~projected/kube-api-access-zdhn7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4ebbfa3-4272-4095-8a57-a55146be5135/volumes/kubernetes.io~projected/kube-api-access-s62l8:{mountpoint:/var/lib/kubelet/pods/a4ebbfa3-4272-4095-8a57-a55146be5135/volumes/kubernetes.io~projected/kube-api-access-s62l8 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a63e5bbb-4666-48a9-8530-64848d5bf70b/volumes/kubernetes.io~projected/kube-api-access-q7dh4:{mountpoint:/var/lib/kubelet/pods/a63e5bbb-4666-48a9-8530-64848d5bf70b/volumes/kubernetes.io~projected/kube-api-access-q7dh4 major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b070df60-6cdb-43fa-842b-ca11d66b1770/volumes/kubernetes.io~projected/kube-api-access-hpbfd:{mountpoint:/var/lib/kubelet/pods/b070df60-6cdb-43fa-842b-ca11d66b1770/volumes/kubernetes.io~projected/kube-api-access-hpbfd major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~projected/kube-api-access-cwtms:{mountpoint:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~projected/kube-api-access-cwtms major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/volumes/kubernetes.io~projected/kube-api-access-mrfjn:{mountpoint:/var/lib/kubelet/pods/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/volumes/kubernetes.io~projected/kube-api-access-mrfjn major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5013b2f-c423-4c36-aaa0-917c35cf23f8/volumes/kubernetes.io~projected/kube-api-access-hrrh4:{mountpoint:/var/lib/kubelet/pods/c5013b2f-c423-4c36-aaa0-917c35cf23f8/volumes/kubernetes.io~projected/kube-api-access-hrrh4 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5013b2f-c423-4c36-aaa0-917c35cf23f8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c5013b2f-c423-4c36-aaa0-917c35cf23f8/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cec9279a-2cd4-49bd-823c-398f6528bbe7/volumes/kubernetes.io~projected/kube-api-access-whzcb:{mountpoint:/var/lib/kubelet/pods/cec9279a-2cd4-49bd-823c-398f6528bbe7/volumes/kubernetes.io~projected/kube-api-access-whzcb major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cec9279a-2cd4-49bd-823c-398f6528bbe7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cec9279a-2cd4-49bd-823c-398f6528bbe7/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf8a23a0-a643-42bd-84f7-29a2bb9138b7/volumes/kubernetes.io~projected/kube-api-access-mpbfg:{mountpoint:/var/lib/kubelet/pods/cf8a23a0-a643-42bd-84f7-29a2bb9138b7/volumes/kubernetes.io~projected/kube-api-access-mpbfg major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225/volumes/kubernetes.io~projected/kube-api-access-t9bzx:{mountpoint:/var/lib/kubelet/pods/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225/volumes/kubernetes.io~projected/kube-api-access-t9bzx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~projected/kube-api-access-qfbcm:{mountpoint:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~projected/kube-api-access-qfbcm major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da62aabf-4741-4ada-b763-f5d62d142ae4/volumes/kubernetes.io~projected/kube-api-access-mr5qs:{mountpoint:/var/lib/kubelet/pods/da62aabf-4741-4ada-b763-f5d62d142ae4/volumes/kubernetes.io~projected/kube-api-access-mr5qs major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3746ecb-830d-4d86-8400-21f4d2985b22/volumes/kubernetes.io~projected/kube-api-access-297ff:{mountpoint:/var/lib/kubelet/pods/e3746ecb-830d-4d86-8400-21f4d2985b22/volumes/kubernetes.io~projected/kube-api-access-297ff major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e40c5726-14a1-4f26-8d5f-218b4ff946a8/volumes/kubernetes.io~projected/kube-api-access-lc72p:{mountpoint:/var/lib/kubelet/pods/e40c5726-14a1-4f26-8d5f-218b4ff946a8/volumes/kubernetes.io~projected/kube-api-access-lc72p major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea88fbd7-988d-44a5-a12e-fd5e64e8da79/volumes/kubernetes.io~projected/kube-api-access-kpm8x:{mountpoint:/var/lib/kubelet/pods/ea88fbd7-988d-44a5-a12e-fd5e64e8da79/volumes/kubernetes.io~projected/kube-api-access-kpm8x major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec657dda-01bf-42d4-96de-1f9bb920dfff/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ec657dda-01bf-42d4-96de-1f9bb920dfff/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volumes/kubernetes.io~projected/kube-api-access-nj5wd:{mountpoint:/var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volumes/kubernetes.io~projected/kube-api-access-nj5wd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f8730c7a8df3faf85ad821e31a94d024783b5a3147c811578641c60dd6d17876/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cd64311873d94429e87ae4d48181dbbea75841bef81ae21a330cf9a7f88caf11/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3f7263f3c7aa3042486a09780cef9a816548bced155419293e8284ff74db5d76/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/57a82f8cedfec9ad64524531c22da60f1a74514d71c0022ef29d0ba95468fcd2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d3be42ff221a727cf1ed4cb658883141aabbe311c3182383bd6f0c521a2bea64/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a227e0072f76aa83695a3aed5603229c187561e63ee51efb27f8ea25d8c2235c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0a0dea6bc06d278f4a054433d95ec8be0775fc63ae0ea33a8cdb70ff4fc4e2e6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6e81d8ad8faead01f61cd6ccb144537e4b1893cb162cb36b06ed45968ecc8ee9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/64233faffc6088e9e3887b7f333381ef7cb1a02fe0b04ff8fd831178aac47ef9/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/29a118ff41356003cf6772d1b02211cc323276579e6d696d6c0377bae0457398/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/f05feedfa65b5015c8aba2bcd1fc3397b347c2980c0509867abf3d98a5264626/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8e9b9d326e9098c9245c103bcfa16bae2c553922b28cd7895c82e5b8465079a8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/2a8de64141fe4211bfd31bf1841af7994c8223d9ab9e6748d337180018d1b91f/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4786dac2d69c0adb0b86166395e536da5a323b654be2bdbcc2da921396c6511d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6d0fd4bb05adb5665df836c68dfb0beccd99d5dc16366c6c7067ec1911b83d6e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/83f608870ef49a87e8cb6bca68675a05634f7722ab538a65a3f46914a65e057d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f9d2df6b7584923683097eba920816fbfd76c6d26d676aed75c384588984e7af/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cc41bb7dde1c4276b37c873081b3f03e4994a0b916f13cefe59ac1e962affd7f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e904263b07f9cb087577f5183182a41d1b08d1bfc129ef0d804cb6f5957b725c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e09778356999b6016cdb69fbc3baba23562c1f00b0a4d40147561834961be343/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2bd31cee76c9ffad35f407ee6b88f8a488f2eb9bf4cb2e7f779f9fb2bb4f52e3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/82c81954d94d4189234d131205f3e906138b60b99226b1c49b2cec5b2c58538b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/04ecba6889ee36e6dc406a60bfad5e4c11f7ee8e92dfa04ca9024fc8cc08970d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/35b1d55d2ca3f787109a543bb19d497e2843d579dcfc659a3418c9ae7e519428/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/2cb25f7111355cb6fd06e5fcb52329ed08aedd623df48a8f6089d671d0906634/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8d5548bd8de9cce7dbf4a17066ab9c2202e218318ffbaca513ae024aa1a5909b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/daee704c888edb9ada8a6b2812ef7c9f277fa495cbee796659b02eda9bd9b401/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d2c42ed71eb877f8c9d65152d4c21805641e5343855ea4b907654b0f431bb890/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e57094be0a2a66f714a4b5e641fc718ccb54f93c1c6033de59e8e61beeae7a83/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/0edfd2fd6974025d926ba9d2f52473de46c76bba7b083c31c29b9791ef1da292/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7cf0886ac479c966d5710d677cb617aa667b7deef49e9ba1c3cf0c2499fe02e6/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/38dab91a3657f36c414eba838176eb43ffbf7ec2ea98aa4dbd97ff4c2ff1310d/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/0037f6958cd58153c71e736b7667cac83491ac6082aeec5ed9e0c1a88f072a65/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/9985e77b6b1c8b3e5276f1c2c58d8eba11a5e94b5831169d14dcbb9f67b441b5/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/400355fd37b8acdfb480ca8f317670ed62cf8dfe7602b9d335ea4b28780d4951/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/828836b520eaa59491138a51b5f8835815b3d4d43fb107073ab01cee0ad8dbf7/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/a9d6e1477493e5e7ae9fad7bc6e818715ad5840a48b931826a37f4e0607477a2/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/9846bdfedf79a11ec11ca241321749532d4fc877a8e80b4d2743179437ba4b6d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/d149d14afe499396c9c98b2cea2af27fffdf68df2944afdacb866f45efef3d96/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/9af78463dbd328bfdc9c9169040e21138fc6bad28e8013e6d771484f0b66f769/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/e5a9d362724ef6918bf147e975101f2623f22782251bd7073bf8baafcb01fc58/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/c1a3f5d996b6ee649731aacfa51000a426f382304962b49a6a437c73aae9086a/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/04e580556793d6b57b6516cf97948cf9eb383ea25eaa5d378b8775cecb3e635d/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/17c377a4bc00c5f34017a2bc57675825eb8439da3ed18d2d177add4d535c2cdf/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/36e3351d6977520f8f526170b2679dc546726d8a28451244237fb132f70b0f33/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/1ef431a1461d06069ea032d8de6ac85af96e782c8f9656e6223c670451fad95c/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/21c56bbab368dbf7301bf7e4f37dcb1ae15c93b950507014abe564bdf82f4502/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/425b4efb9780cda47248febe1e46cd6b3af980d243d4a0398e1127d2a1604af4/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/74e882e28c94014219a870b326382282db4d8a60bcedfeb7903502721a8323b6/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/44db8fdec0d794de34c115ff7e7ac342f1628806b6e995193fc34afff923a317/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3ede35b591b0af2d188d6fb34dbca75d42f6451b2d877e675f77732ab41a0751/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/04df3c92b8d30e0c0144ffc85f73a80386c0181c87781d7a7cc1b74f2986a611/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/35361440319f5860a3ccb60d39d21c91ce6162227ae180cf325f5b442d6cd5e0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/653516ab0107e61342be0954e8178373f5ee99dba7dd7a28841ecf6c119918b7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/850d17308f4b3af7a18ffab3d7acfc0a3f698f101680495e96a73d997b15b611/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2a6377bc893e93d48b3d193a63bc5fc9b734ad0d9875fdae2c082d2d2cdf00f4/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/01c86c2ec98512e968352a4e4b09040cc0f07909d3a198bb28ac81acc369acbe/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7a6ba2ee6cf944f0579a974ebbf36ebe5776345953cc770ed385c78d1fc85962/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0d457e6327f92a742cc261839f2078013a71b36a8c10f3806005441407824422/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/684762aab52a379e62a0f58f49dd6fa34eb9cbe5b90cceb1b364e75485ed66b6/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.277885    5060 manager.go:217] Machine: {Timestamp:2026-10-30 11:29:56.277410817 +0000 UTC m=+0.072295302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100178 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24de79abcef615b90fbe35c38b7f10 SystemUUID:ec24de79-abce-f615-b90f-be35c38b7f10 BootID:f121c739-4714-4e7c-b546-f9675e87a63b Filesystems:[{Device:/var/lib/kubelet/pods/da62aabf-4741-4ada-b763-f5d62d142ae4/volumes/kubernetes.io~projected/kube-api-access-mr5qs DeviceMajor:0 DeviceMinor:310 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec657dda-01bf-42d4-96de-1f9bb920dfff/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cec9279a-2cd4-49bd-823c-398f6528bbe7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~projected/kube-api-access-qfbcm DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91cfe548-7bc2-4679-99a3-7212bdc92193/volumes/kubernetes.io~projected/kube-api-access-zdhn7 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a17a4c9fce5a49565c323d17bba2031cc7d9b3ed141fd3eca3ac438a3046604f/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09e61154baaa33668861550255f0d0bbc36c472fe0608e468bee236ba93b8b8d/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8281e846591a375204a867b18c6cd2f19742612898677ea1844c9cc33418ca5a/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23360850-74e3-473e-a579-70d4fcc5a9d0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e40c5726-14a1-4f26-8d5f-218b4ff946a8/volumes/kubernetes.io~projected/kube-api-access-lc72p DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3746ecb-830d-4d86-8400-21f4d2985b22/volumes/kubernetes.io~projected/kube-api-access-297ff DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1255d8a23fb23bd450dcafa6722e054f47a68521a7ac6c8e915cd6edf30c546d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cf8a23a0-a643-42bd-84f7-29a2bb9138b7/volumes/kubernetes.io~projected/kube-api-access-mpbfg DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8416193c666513dd32ebf676612db1234fa08ded73908ee249bac2aa104d91c1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/23360850-74e3-473e-a579-70d4fcc5a9d0/volumes/kubernetes.io~projected/kube-api-access-6gvld DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5013b2f-c423-4c36-aaa0-917c35cf23f8/volumes/kubernetes.io~projected/kube-api-access-hrrh4 DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb51ace574c4124f9741c13966cac52a3267c698404dca63badcb3e083223091/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4ebbfa3-4272-4095-8a57-a55146be5135/volumes/kubernetes.io~projected/kube-api-access-s62l8 DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f866f9660dc1220ae7f13303d33fb85c561016336fc67977c92c1ab38c9b4cc6/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/52a6b071-d9c7-4d71-aeae-65e818f74dec/volumes/kubernetes.io~projected/kube-api-access-q4mfs DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8071afea22a0b00e0a05bdd51ebca64a5785321a63278cf8698190acff56639e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~projected/kube-api-access-bjcb6 DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eace550a8122d8e5442d442d7d44500a94dbf4a9e903c1de24573a0805612df/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5339a2af-2f4a-4444-b146-dcb849fa7449/volumes/kubernetes.io~projected/kube-api-access-hzwtx DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225/volumes/kubernetes.io~projected/kube-api-access-t9bzx DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdec2495831e4bfd4fcb01829e99a09537757687d0329da2707dced1d7190353/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/977868089cb1b4a8af5b255b917cfb8ae68cacfcff34a7e79a07bd1023f42c25/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~projected/kube-api-access-cwtms DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a63e5bbb-4666-48a9-8530-64848d5bf70b/volumes/kubernetes.io~projected/kube-api-access-q7dh4 DeviceMajor:0 DeviceMinor:321 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1f5f4127-5c51-4029-8ba2-0f5c01bad573/volumes/kubernetes.io~projected/kube-api-access-lvg6b DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22a93e48-3382-47af-adef-2ceb0d1fc6aa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volumes/kubernetes.io~projected/kube-api-access-nj5wd DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea88fbd7-988d-44a5-a12e-fd5e64e8da79/volumes/kubernetes.io~projected/kube-api-access-kpm8x DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38a165ac547d5c981690b088c5b5d0c4f262adb5cd1212f0120aabe216164fef/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d000fdc-e5ed-4fce-bb1e-6715932a37b1/volumes/kubernetes.io~projected/kube-api-access-b59qz DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c980147-d63b-436d-bff6-5a458249fe5a/volumes/kubernetes.io~projected/kube-api-access-lb779 DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f069f0a4649e58ee2fb27e1da0177a0dbffe3aac104908feba01cd075d1edfce/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b038abec5b851cb1ca448b57473afa2242dad7610a6d70fd0af8adba0199274d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0a3c15b5d21f37a66d2204f17a2127730fe9160c3f9fd8a3a74bb4561110dcc/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5815b6d3-0bdd-45b2-a673-5f493146c0f8/volumes/kubernetes.io~projected/kube-api-access-jbrt9 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/46099da2-dc06-45b7-bc3c-139b84c896e0/volumes/kubernetes.io~projected/kube-api-access-v7jdn DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8d1c894a07c4817863b55c70431e94dfd6620276a4d46683a34bfe7d2ec88b8/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c68e591-1f33-4482-9843-d0412cec6bce/volumes/kubernetes.io~projected/kube-api-access-nr5b8 DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fb42d5bc82ac44d8a7ea19edb6da77f10a33f706de06d305fb792c588b77a74/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ed2ea1feb163f20035ce2e201cc23c62c9b57980b77172d76b1b41ec05a559d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fddeddb5-22c9-4d0b-a77f-6d2a12319856/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f4370e6-112e-4d65-940f-5ca5263ce913/volumes/kubernetes.io~projected/kube-api-access-gg5n6 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c4adbf03d7af5dfe077b96ac20b5b4b81f1d0b1f1ead66fdccd24507ae607b0/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63257d8b-a976-4db1-8721-98f7d7fefed8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b070df60-6cdb-43fa-842b-ca11d66b1770/volumes/kubernetes.io~projected/kube-api-access-hpbfd DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14ba76bf-37ab-43fd-a583-5a9343e75964/volumes/kubernetes.io~projected/kube-api-access-tbt8b DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d1833122f748ec1de3399d6c07e845dd2b369d26e6cb671035d49b040097a87/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5013b2f-c423-4c36-aaa0-917c35cf23f8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6dbb0f04a2983bec45887960233be09029ab8419a07f74d1c4c351283b9ecb5/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d1d8529-d285-41d3-b179-6f3343059567/volumes/kubernetes.io~projected/kube-api-access-bmzlw DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22a93e48-3382-47af-adef-2ceb0d1fc6aa/volumes/kubernetes.io~projected/kube-api-access-5wzgg DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/volumes/kubernetes.io~projected/kube-api-access-mrfjn DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cec9279a-2cd4-49bd-823c-398f6528bbe7/volumes/kubernetes.io~projected/kube-api-access-whzcb DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c714f27-241c-4c7f-86fe-2c3eb89d0542/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fea51a9fb35969560abfbf41c22a129a1fd4005827aed85e264f93ecd7860978/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:09e61154baaa336 MacAddress:36:20:54:d0:8a:fa Speed:10000 Mtu:8901} {Name:1c4adbf03d7af5d MacAddress:aa:e0:54:88:9c:54 Speed:10000 Mtu:8901} {Name:38a165ac547d5c9 MacAddress:ca:94:f1:d7:07:6d Speed:10000 Mtu:8901} {Name:38bc9cd8c4f142f MacAddress:1a:24:ad:54:b9:93 Speed:10000 Mtu:8901} {Name:3ed2ea1feb163f2 MacAddress:d6:c8:87:45:10:30 Speed:10000 Mtu:8901} {Name:6fb42d5bc82ac44 MacAddress:56:66:b4:b0:9d:fc Speed:10000 Mtu:8901} {Name:8071afea22a0b00 MacAddress:96:4a:1d:d8:16:81 Speed:10000 Mtu:8901} {Name:8281e846591a375 MacAddress:4e:32:fc:50:20:78 Speed:10000 Mtu:8901} {Name:977868089cb1b4a MacAddress:4e:4d:3e:e0:37:cf Speed:10000 Mtu:8901} {Name:9eace550a8122d8 MacAddress:d6:0d:4e:df:83:6d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:7d:64:b3 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:10:9d:3e:2d:6f Speed:0 Mtu:8901} {Name:c6dbb0f04a2983b MacAddress:ca:c1:de:fa:a3:17 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:7d:64:b3 Speed:0 Mtu:9001} {Name:f069f0a4649e58e MacAddress:7e:e1:3b:ef:27:7f Speed:10000 Mtu:8901} {Name:f0a3c15b5d21f37 MacAddress:4a:94:7a:3a:51:37 Speed:10000 Mtu:8901} {Name:f866f9660dc1220 MacAddress:7e:ca:c1:e9:bf:8c Speed:10000 Mtu:8901} {Name:fea51a9fb359695 MacAddress:76:05:d8:a2:9b:45 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ba:d9:d7:8b:5d:bf Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:da:0f:2c:c1:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278232    5060 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278277    5060 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278535    5060 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278552    5060 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278681    5060 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278689    5060 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278699    5060 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278718    5060 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278735    5060 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278873    5060 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.278970    5060 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279001    5060 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279014    5060 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279024    5060 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279033    5060 kubelet.go:405] "Adding apiserver pod source"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279041    5060 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279795    5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 11:29:56.280149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.279804    5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:29:56.281297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.280306    5060 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 11:29:56.282707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.282687    5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 11:29:56.282707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.282701    5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:29:56.282853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.282814    5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 11:29:56.283014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283000    5060 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 11:29:56.283070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283029    5060 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 11:29:56.283179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283165    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 11:29:56.283228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283196    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 11:29:56.283228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283210    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 11:29:56.283228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283217    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 11:29:56.283228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283222    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 11:29:56.283228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283228    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283233    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283243    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283250    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283256    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283272    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283283    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283309    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 11:29:56.283484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.283318    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 11:29:56.286832 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.286819    5060 server.go:1263] "Started kubelet"
Oct 30 11:29:56.286969 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.286906    5060 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 11:29:56.287077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.287019    5060 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 11:29:56.287122 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.287106    5060 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 11:29:56.287376 ip-10-0-142-105 systemd[1]: Started Kubernetes Kubelet.
Oct 30 11:29:56.288603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.287584    5060 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 11:29:56.294160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.291927    5060 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 11:29:56.294160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.292143    5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 11:29:56.294160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.293533    5060 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 11:29:56.294160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.293552    5060 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 11:29:56.294160 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.293568    5060 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Oct 30 11:29:56.294160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.293707    5060 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 11:29:56.297172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.297125    5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:24:17 +0000 UTC" deadline="2028-05-24 01:21:56.047526801 +0000 UTC"
Oct 30 11:29:56.297950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.297931    5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13717h51m59.749603016s"
Oct 30 11:29:56.298080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.297627    5060 factory.go:55] Registering systemd factory
Oct 30 11:29:56.298178 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.298086    5060 factory.go:223] Registration of the systemd container factory successfully
Oct 30 11:29:56.298178 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.298128    5060 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 11:29:56.300188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300133    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec9279a-2cd4-49bd-823c-398f6528bbe7" volumeName="kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-service-ca-bundle" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300196    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300214    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f4370e6-112e-4d65-940f-5ca5263ce913" volumeName="kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300230    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22a93e48-3382-47af-adef-2ceb0d1fc6aa" volumeName="kubernetes.io/projected/22a93e48-3382-47af-adef-2ceb0d1fc6aa-kube-api-access-5wzgg" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300247    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a6b071-d9c7-4d71-aeae-65e818f74dec" volumeName="kubernetes.io/projected/52a6b071-d9c7-4d71-aeae-65e818f74dec-kube-api-access-q4mfs" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300258    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5339a2af-2f4a-4444-b146-dcb849fa7449" volumeName="kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-cni-binary-copy" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300271    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c714f27-241c-4c7f-86fe-2c3eb89d0542" volumeName="kubernetes.io/projected/5c714f27-241c-4c7f-86fe-2c3eb89d0542-kube-api-access-bjcb6" seLinuxMountContext=""
Oct 30 11:29:56.300289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300284    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" volumeName="kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-trusted-ca" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300300    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddeddb5-22c9-4d0b-a77f-6d2a12319856" volumeName="kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-env-overrides" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300314    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46099da2-dc06-45b7-bc3c-139b84c896e0" volumeName="kubernetes.io/projected/46099da2-dc06-45b7-bc3c-139b84c896e0-kube-api-access-v7jdn" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300325    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c714f27-241c-4c7f-86fe-2c3eb89d0542" volumeName="kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-stats-auth" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300337    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300349    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14ba76bf-37ab-43fd-a583-5a9343e75964" volumeName="kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-tuned" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300360    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c980147-d63b-436d-bff6-5a458249fe5a" volumeName="kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300376    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5339a2af-2f4a-4444-b146-dcb849fa7449" volumeName="kubernetes.io/projected/5339a2af-2f4a-4444-b146-dcb849fa7449-kube-api-access-hzwtx" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300387    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5815b6d3-0bdd-45b2-a673-5f493146c0f8" volumeName="kubernetes.io/projected/5815b6d3-0bdd-45b2-a673-5f493146c0f8-kube-api-access-jbrt9" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300399    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1d8529-d285-41d3-b179-6f3343059567" volumeName="kubernetes.io/projected/5d1d8529-d285-41d3-b179-6f3343059567-kube-api-access-bmzlw" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300410    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cfe548-7bc2-4679-99a3-7212bdc92193" volumeName="kubernetes.io/projected/91cfe548-7bc2-4679-99a3-7212bdc92193-kube-api-access-zdhn7" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300421    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" volumeName="kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-config" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300432    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" volumeName="kubernetes.io/projected/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-kube-api-access-mrfjn" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300443    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23360850-74e3-473e-a579-70d4fcc5a9d0" volumeName="kubernetes.io/secret/23360850-74e3-473e-a579-70d4fcc5a9d0-serving-cert" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300471    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4ebbfa3-4272-4095-8a57-a55146be5135" volumeName="kubernetes.io/configmap/a4ebbfa3-4272-4095-8a57-a55146be5135-telemetry-config" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300485    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec9279a-2cd4-49bd-823c-398f6528bbe7" volumeName="kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-tmp" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300496    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec9279a-2cd4-49bd-823c-398f6528bbe7" volumeName="kubernetes.io/projected/cec9279a-2cd4-49bd-823c-398f6528bbe7-kube-api-access-whzcb" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300509    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300520    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da62aabf-4741-4ada-b763-f5d62d142ae4" volumeName="kubernetes.io/projected/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-api-access-mr5qs" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300541    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4ebbfa3-4272-4095-8a57-a55146be5135" volumeName="kubernetes.io/projected/a4ebbfa3-4272-4095-8a57-a55146be5135-kube-api-access-s62l8" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300555    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c173a778-6503-4ebc-9bd4-750a98b0f9d9" volumeName="kubernetes.io/configmap/c173a778-6503-4ebc-9bd4-750a98b0f9d9-nginx-conf" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300571    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300583    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40c5726-14a1-4f26-8d5f-218b4ff946a8" volumeName="kubernetes.io/projected/e40c5726-14a1-4f26-8d5f-218b4ff946a8-kube-api-access-lc72p" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300593    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea88fbd7-988d-44a5-a12e-fd5e64e8da79" volumeName="kubernetes.io/projected/ea88fbd7-988d-44a5-a12e-fd5e64e8da79-kube-api-access-kpm8x" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300612    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22a93e48-3382-47af-adef-2ceb0d1fc6aa" volumeName="kubernetes.io/secret/22a93e48-3382-47af-adef-2ceb0d1fc6aa-cert" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300623    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225" volumeName="kubernetes.io/configmap/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-iptables-alerter-script" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300636    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14ba76bf-37ab-43fd-a583-5a9343e75964" volumeName="kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-tmp" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300649    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5f4127-5c51-4029-8ba2-0f5c01bad573" volumeName="kubernetes.io/projected/1f5f4127-5c51-4029-8ba2-0f5c01bad573-kube-api-access-lvg6b" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300661    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c980147-d63b-436d-bff6-5a458249fe5a" volumeName="kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300672    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a63e5bbb-4666-48a9-8530-64848d5bf70b" volumeName="kubernetes.io/projected/a63e5bbb-4666-48a9-8530-64848d5bf70b-kube-api-access-q7dh4" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300686    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddeddb5-22c9-4d0b-a77f-6d2a12319856" volumeName="kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300697    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300713    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c980147-d63b-436d-bff6-5a458249fe5a" volumeName="kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300724    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5339a2af-2f4a-4444-b146-dcb849fa7449" volumeName="kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-daemon-config" seLinuxMountContext=""
Oct 30 11:29:56.300719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300736    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec9279a-2cd4-49bd-823c-398f6528bbe7" volumeName="kubernetes.io/secret/cec9279a-2cd4-49bd-823c-398f6528bbe7-serving-cert" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300748    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300759    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddeddb5-22c9-4d0b-a77f-6d2a12319856" volumeName="kubernetes.io/projected/fddeddb5-22c9-4d0b-a77f-6d2a12319856-kube-api-access-nj5wd" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300771    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f4370e6-112e-4d65-940f-5ca5263ce913" volumeName="kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-binary-copy" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300782    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f4370e6-112e-4d65-940f-5ca5263ce913" volumeName="kubernetes.io/projected/0f4370e6-112e-4d65-940f-5ca5263ce913-kube-api-access-gg5n6" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300793    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63257d8b-a976-4db1-8721-98f7d7fefed8" volumeName="kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300804    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b070df60-6cdb-43fa-842b-ca11d66b1770" volumeName="kubernetes.io/projected/b070df60-6cdb-43fa-842b-ca11d66b1770-kube-api-access-hpbfd" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300815    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300825    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300836    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14ba76bf-37ab-43fd-a583-5a9343e75964" volumeName="kubernetes.io/projected/14ba76bf-37ab-43fd-a583-5a9343e75964-kube-api-access-tbt8b" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300848    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec9279a-2cd4-49bd-823c-398f6528bbe7" volumeName="kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300859    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec9279a-2cd4-49bd-823c-398f6528bbe7" volumeName="kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-snapshots" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300877    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec657dda-01bf-42d4-96de-1f9bb920dfff" volumeName="kubernetes.io/secret/ec657dda-01bf-42d4-96de-1f9bb920dfff-agent-certs" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300889    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddeddb5-22c9-4d0b-a77f-6d2a12319856" volumeName="kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-config" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300900    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddeddb5-22c9-4d0b-a77f-6d2a12319856" volumeName="kubernetes.io/secret/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300920    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" volumeName="kubernetes.io/secret/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-serving-cert" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300932    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5013b2f-c423-4c36-aaa0-917c35cf23f8" volumeName="kubernetes.io/configmap/c5013b2f-c423-4c36-aaa0-917c35cf23f8-config" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300944    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5013b2f-c423-4c36-aaa0-917c35cf23f8" volumeName="kubernetes.io/projected/c5013b2f-c423-4c36-aaa0-917c35cf23f8-kube-api-access-hrrh4" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300957    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d000fdc-e5ed-4fce-bb1e-6715932a37b1" volumeName="kubernetes.io/projected/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-kube-api-access-b59qz" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300970    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23360850-74e3-473e-a579-70d4fcc5a9d0" volumeName="kubernetes.io/configmap/23360850-74e3-473e-a579-70d4fcc5a9d0-config" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300981    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.300994    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301006    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301018    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c980147-d63b-436d-bff6-5a458249fe5a" volumeName="kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301029    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46099da2-dc06-45b7-bc3c-139b84c896e0" volumeName="kubernetes.io/configmap/46099da2-dc06-45b7-bc3c-139b84c896e0-serviceca" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301041    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5013b2f-c423-4c36-aaa0-917c35cf23f8" volumeName="kubernetes.io/secret/c5013b2f-c423-4c36-aaa0-917c35cf23f8-serving-cert" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301052    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8a23a0-a643-42bd-84f7-29a2bb9138b7" volumeName="kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301065    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da62aabf-4741-4ada-b763-f5d62d142ae4" volumeName="kubernetes.io/empty-dir/da62aabf-4741-4ada-b763-f5d62d142ae4-data-volume" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301078    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d000fdc-e5ed-4fce-bb1e-6715932a37b1" volumeName="kubernetes.io/configmap/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-config-volume" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301090    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d000fdc-e5ed-4fce-bb1e-6715932a37b1" volumeName="kubernetes.io/empty-dir/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-tmp-dir" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301101    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225" volumeName="kubernetes.io/projected/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-kube-api-access-t9bzx" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301112    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58c2f33-be41-4e16-91bc-293ee27d908e" volumeName="kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301125    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec657dda-01bf-42d4-96de-1f9bb920dfff" volumeName="kubernetes.io/configmap/ec657dda-01bf-42d4-96de-1f9bb920dfff-konnectivity-ca" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301158    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5f4127-5c51-4029-8ba2-0f5c01bad573" volumeName="kubernetes.io/configmap/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-config" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301173    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5815b6d3-0bdd-45b2-a673-5f493146c0f8" volumeName="kubernetes.io/empty-dir/5815b6d3-0bdd-45b2-a673-5f493146c0f8-tmp-dir" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301186    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3746ecb-830d-4d86-8400-21f4d2985b22" volumeName="kubernetes.io/projected/e3746ecb-830d-4d86-8400-21f4d2985b22-kube-api-access-297ff" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301198    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f4370e6-112e-4d65-940f-5ca5263ce913" volumeName="kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301211    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23360850-74e3-473e-a579-70d4fcc5a9d0" volumeName="kubernetes.io/projected/23360850-74e3-473e-a579-70d4fcc5a9d0-kube-api-access-6gvld" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301224    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c714f27-241c-4c7f-86fe-2c3eb89d0542" volumeName="kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-default-certificate" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301236    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c68e591-1f33-4482-9843-d0412cec6bce" volumeName="kubernetes.io/projected/8c68e591-1f33-4482-9843-d0412cec6bce-kube-api-access-nr5b8" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301249    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301260    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bebf069a-0c3e-44c6-9ad8-23dc0056c442" volumeName="kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301272    5060 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.301278    5060 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.302020    5060 factory.go:153] Registering CRI-O factory
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.302037    5060 factory.go:223] Registration of the crio container factory successfully
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.302113    5060 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.302137    5060 factory.go:103] Registering Raw factory
Oct 30 11:29:56.302391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.302156    5060 manager.go:1199] Started watching for new ooms in manager
Oct 30 11:29:56.304574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.302589    5060 manager.go:319] Starting recovery of all containers
Oct 30 11:29:56.315175 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.315147    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/1.log"
Oct 30 11:29:56.315175 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.315160    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/1.log"
Oct 30 11:29:56.324837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.324805    5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 11:29:56.326360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.326342    5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 11:29:56.326360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.326361    5060 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 11:29:56.326590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.326381    5060 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 11:29:56.326590 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.326415    5060 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 11:29:56.331796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.331769    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/1.log"
Oct 30 11:29:56.331914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.331804    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572" exitCode=255
Oct 30 11:29:56.346002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.345938    5060 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 11:29:56.346002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.345945    5060 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:56.346192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.346050    5060 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:56.358923 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.358883    5060 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:56.359155 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.359135    5060 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:56.365221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.365182    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="8afefd9c6a9d2a5799e281eccb81e22faae141a9bef698cada7e237034766920" exitCode=0
Oct 30 11:29:56.365379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.365223    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="9acaeaf4009561452d897851adfed5a5a14b5fed37c14c64057505484bf9a7be" exitCode=0
Oct 30 11:29:56.365379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.365232    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="d5c2b484b4d6e11c22abbe557dfd7026a4e2123e1e655cde85a11192c7df6c40" exitCode=0
Oct 30 11:29:56.365379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.365240    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="8e051263ba5ddaa03513ee31c2b31afbaf37532efac55cc8d78374e2e78a9916" exitCode=0
Oct 30 11:29:56.365379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.365247    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="b180b1661dc4482ee851b121f37bd64741aac7347e27b97b82c2ea470598fcaa" exitCode=0
Oct 30 11:29:56.365379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.365254    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f4370e6-112e-4d65-940f-5ca5263ce913" containerID="a9d950ab1c27de249032c3e5b21cccc71b1144194fd6e70661e9185d5d585bf8" exitCode=0
Oct 30 11:29:56.367161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.367135    5060 generic.go:356] "Generic (PLEG): container finished" podID="1793a91293e97f2ab4b51edb090789a9" containerID="936382754aeb57f1d7d23f2eb4525fde9c54c8f70b1eecc4b8a2db3d144630bf" exitCode=0
Oct 30 11:29:56.402212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.402190    5060 manager.go:324] Recovery completed
Oct 30 11:29:56.423030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.422995    5060 cpu_manager.go:226] "Starting" policy="none"
Oct 30 11:29:56.423030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423014    5060 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 11:29:56.423030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423036    5060 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423193    5060 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423207    5060 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423233    5060 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423241    5060 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423249    5060 policy_none.go:50] "Start"
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423260    5060 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 11:29:56.423312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423273    5060 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 11:29:56.423642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423469    5060 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 11:29:56.423642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423481    5060 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 11:29:56.423642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.423489    5060 policy_none.go:44] "Start"
Oct 30 11:29:56.426704 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.426679    5060 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 11:29:56.428029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.428005    5060 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 11:29:56.430568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.430554    5060 manager.go:341] "Starting Device Plugin manager"
Oct 30 11:29:56.431034 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.431017    5060 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 11:29:56.431034 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431035    5060 server.go:87] "Starting device plugin registration server"
Oct 30 11:29:56.431318 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431304    5060 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 11:29:56.431383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431320    5060 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 11:29:56.431429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431394    5060 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 11:29:56.431558 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431529    5060 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 11:29:56.431627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431574    5060 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 11:29:56.431627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431581    5060 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 11:29:56.431706 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.431631    5060 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:29:56.431556803Z","UUID":"4670b01a-f0be-4e9e-842f-b7d65fdfe617","Handler":null,"Name":"","Endpoint":""}
Oct 30 11:29:56.433247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.433195    5060 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 11:29:56.433247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.433223    5060 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 11:29:56.433247 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.433235    5060 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 11:29:56.531972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.531939    5060 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:56.534896 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.534869    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:56.535014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.534910    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:56.535014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.534925    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:56.535014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.534979    5060 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.573213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.573187    5060 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.573359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.573269    5060 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.627608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.627512    5060 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"]
Oct 30 11:29:56.628182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628167    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.628257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628158    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"1793a91293e97f2ab4b51edb090789a9","Type":"ContainerStarted","Data":"c2ef2193b6dcd8a1995bab13164647f8764810e309d25978d6989881c49381ff"}
Oct 30 11:29:56.628316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628296    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"1793a91293e97f2ab4b51edb090789a9","Type":"ContainerDied","Data":"936382754aeb57f1d7d23f2eb4525fde9c54c8f70b1eecc4b8a2db3d144630bf"}
Oct 30 11:29:56.628361 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628328    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"1793a91293e97f2ab4b51edb090789a9","Type":"ContainerStarted","Data":"43a274e1a0c3d2dc1e57c402601d6b051b23e78892979ba633b28e95427fe4b7"}
Oct 30 11:29:56.628361 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628339    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.628361 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628352    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" event={"ID":"148cbd5f3b955e9e79f6d6f208c89d56","Type":"ContainerStarted","Data":"834ea26f3c0ced8af9b10a7cb6ba6597a78b92b4586251b181660ea8b2da710c"}
Oct 30 11:29:56.628440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.628365    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" event={"ID":"148cbd5f3b955e9e79f6d6f208c89d56","Type":"ContainerStarted","Data":"0859a565a6a62ac07bdf0a867f89e6d019a702dad65bca7394df2dd09313a87a"}
Oct 30 11:29:56.651954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.651932    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:56.652058 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.652009    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.652523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.652502    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:56.652629 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:56.652546    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.747842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.747811    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/148cbd5f3b955e9e79f6d6f208c89d56-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"148cbd5f3b955e9e79f6d6f208c89d56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.747842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.747842    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.748024 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.747861    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.848401 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.848367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.848581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.848403    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.848581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.848429    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/148cbd5f3b955e9e79f6d6f208c89d56-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"148cbd5f3b955e9e79f6d6f208c89d56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.848581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.848498    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/148cbd5f3b955e9e79f6d6f208c89d56-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"148cbd5f3b955e9e79f6d6f208c89d56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.848581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.848499    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:56.848581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:56.848498    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1793a91293e97f2ab4b51edb090789a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"1793a91293e97f2ab4b51edb090789a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:57.280157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.280069    5060 apiserver.go:52] "Watching apiserver"
Oct 30 11:29:57.334423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.334394    5060 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 11:29:57.335096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.335071    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-488kk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g","kube-system/global-pull-secret-syncer-r4xxg","openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq","openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29","openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7","kube-system/kas-connection-checker-685895c7c9-x2vn2","openshift-cluster-node-tuning-operator/tuned-64dlw","openshift-network-diagnostics/network-check-target-vxtn4","openshift-network-operator/iptables-alerter-vhm89","openshift-image-registry/node-ca-dhfxh","kube-system/kas-connection-checker-685895c7c9-5b47m","openshift-image-registry/image-registry-7f7dc85469-gjnw7","openshift-insights/insights-runtime-extractor-x9lhg","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b","openshift-multus/network-metrics-daemon-6s7j4","openshift-ovn-kubernetes/ovnkube-node-n2zn7","openshift-dns/node-resolver-7lsq8","kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal","openshift-network-console/networking-console-plugin-86c478bb88-b5tmq","openshift-ingress/router-default-7d8f76c5df-xxgbt","openshift-insights/periodic-gathering-t6jts-b62vt","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k","openshift-multus/multus-289w4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w","openshift-image-registry/image-registry-7d654c48b4-zcdvc","openshift-ingress-canary/ingress-canary-pnclc","openshift-insights/insights-operator-7c4986d449-25xnv","openshift-console-operator/console-operator-7cd56b8dbc-scqxd","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz","openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq","openshift-multus/multus-additional-cni-plugins-d2ml5","openshift-dns/dns-default-zgz8g","kube-system/konnectivity-agent-58qds"]
Oct 30 11:29:57.336324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.336303    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.336995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.336974    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.338287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.338267    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.339570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.339551    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.340410 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.340396    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:57.343791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.343766    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.349210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.349192    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.350144 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.350124    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.350723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.350707    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.351039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351022    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.351111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351067    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.351630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351597    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 11:29:57.351630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351604    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 11:29:57.351854 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351802    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 11:29:57.351854 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351821    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 11:29:57.352004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351862    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 11:29:57.352004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351874    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.352004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351879    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.352004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351900    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 11:29:57.352004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.351923    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.352241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.352081    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.352241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.352100    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.352241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.352153    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.364970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.364948    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 11:29:57.389378 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.389359    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:57.389579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.389565    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:57.396504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.396484    5060 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 11:29:57.420303 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.420275    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:57.420433 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.420326    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:57.421614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.421599    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:57.421725 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.421650    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Oct 30 11:29:57.440185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440163    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 11:29:57.440185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440177    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 11:29:57.440335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440169    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 11:29:57.440335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440168    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 11:29:57.440440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440355    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 11:29:57.440557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440534    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dtb5\""
Oct 30 11:29:57.440700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.440687    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 11:29:57.441393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.441367    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 11:29:57.442298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.442276    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k7ct5\""
Oct 30 11:29:57.442553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.442535    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xq7t5\""
Oct 30 11:29:57.442629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.442607    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 11:29:57.442922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.442901    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlvg9\""
Oct 30 11:29:57.443101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443086    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hsk2h\""
Oct 30 11:29:57.443167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443147    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2726q\""
Oct 30 11:29:57.443288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443231    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hnpqt\""
Oct 30 11:29:57.443346 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443289    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmhtb\""
Oct 30 11:29:57.443409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443378    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bltqf\""
Oct 30 11:29:57.443506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443439    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wql9s\""
Oct 30 11:29:57.443563 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443535    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-899rd\""
Oct 30 11:29:57.443625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443609    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6tzdw\""
Oct 30 11:29:57.443727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443704    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hc7n5\""
Oct 30 11:29:57.443872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443839    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 11:29:57.443968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443954    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 11:29:57.444053 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.443960    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k8lzt\""
Oct 30 11:29:57.444149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444129    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-chtrx\""
Oct 30 11:29:57.444210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444194    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 11:29:57.444263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444231    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 11:29:57.444315 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444065    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4s6wd\""
Oct 30 11:29:57.444515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444492    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 11:29:57.444697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444684    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hm9cg\""
Oct 30 11:29:57.444788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444776    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 11:29:57.444925 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.444912    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkn76\""
Oct 30 11:29:57.445023 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.445010    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lfmd8\""
Oct 30 11:29:57.445155 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.445144    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 11:29:57.445300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.445276    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 11:29:57.450043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450025    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.450121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450057    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5wd\" (UniqueName: \"kubernetes.io/projected/fddeddb5-22c9-4d0b-a77f-6d2a12319856-kube-api-access-nj5wd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.450121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450080    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-default-certificate\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.450184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-cnibin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.450184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450163    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm8x\" (UniqueName: \"kubernetes.io/projected/ea88fbd7-988d-44a5-a12e-fd5e64e8da79-kube-api-access-kpm8x\") pod \"kas-connection-checker-685895c7c9-x2vn2\" (UID: \"ea88fbd7-988d-44a5-a12e-fd5e64e8da79\") " pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:57.450254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450190    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcb6\" (UniqueName: \"kubernetes.io/projected/5c714f27-241c-4c7f-86fe-2c3eb89d0542-kube-api-access-bjcb6\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.450254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450214    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-systemd\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.450254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450237    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-sys\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.450395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450263    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297ff\" (UniqueName: \"kubernetes.io/projected/e3746ecb-830d-4d86-8400-21f4d2985b22-kube-api-access-297ff\") pod \"kas-connection-checker-685895c7c9-5b47m\" (UID: \"e3746ecb-830d-4d86-8400-21f4d2985b22\") " pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:57.450395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5013b2f-c423-4c36-aaa0-917c35cf23f8-serving-cert\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:57.450395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450317    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-sys-fs\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.450395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450342    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.450395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450369    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-tuned\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.450395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-default-certificate\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.450637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-tuned\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.450637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450488    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.450637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450509    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5013b2f-c423-4c36-aaa0-917c35cf23f8-serving-cert\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:57.450637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450566    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22a93e48-3382-47af-adef-2ceb0d1fc6aa-cert\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:57.450637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5013b2f-c423-4c36-aaa0-917c35cf23f8-config\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:57.450637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-device-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450638    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450663    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-etc-kubernetes\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450688    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23360850-74e3-473e-a579-70d4fcc5a9d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450730    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-var-lib-kubelet\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46099da2-dc06-45b7-bc3c-139b84c896e0-host\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450782    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22a93e48-3382-47af-adef-2ceb0d1fc6aa-cert\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450802    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-socket-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450836    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450836    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5013b2f-c423-4c36-aaa0-917c35cf23f8-config\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450891    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5b8\" (UniqueName: \"kubernetes.io/projected/8c68e591-1f33-4482-9843-d0412cec6bce-kube-api-access-nr5b8\") pod \"kas-connection-checker-685895c7c9-488kk\" (UID: \"8c68e591-1f33-4482-9843-d0412cec6bce\") " pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:57.450916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450917    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc72p\" (UniqueName: \"kubernetes.io/projected/e40c5726-14a1-4f26-8d5f-218b4ff946a8-kube-api-access-lc72p\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450941    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-host-slash\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.450966    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-kubelet\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451017    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-snapshots\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451035    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-kubernetes\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-tmp\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23360850-74e3-473e-a579-70d4fcc5a9d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451169    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-snapshots\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba76bf-37ab-43fd-a583-5a9343e75964-tmp\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.451262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451249    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.451610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-kubelet\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.451610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451382    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-slash\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.451610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451408    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-cnibin\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.451610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451443    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-stats-auth\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.451610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451502    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.451610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec657dda-01bf-42d4-96de-1f9bb920dfff-konnectivity-ca\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-stats-auth\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451654    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451685    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451708    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451724    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-var-lib-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451740    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.451794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451786    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-iptables-alerter-script\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451822    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4mfs\" (UniqueName: \"kubernetes.io/projected/52a6b071-d9c7-4d71-aeae-65e818f74dec-kube-api-access-q4mfs\") pod \"network-check-source-6f78d59f99-7jfl7\" (UID: \"52a6b071-d9c7-4d71-aeae-65e818f74dec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451852    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451870    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c173a778-6503-4ebc-9bd4-750a98b0f9d9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451870    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451930    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-registration-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.451994    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452018    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452047    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-socket-dir-parent\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.452100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452088    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-k8s-cni-cncf-io\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452153    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-tmp-dir\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452181    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-serving-cert\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-tmp-dir\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452295    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-tmp\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452316    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-serving-cert\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452328    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452347    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-system-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.452400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec9279a-2cd4-49bd-823c-398f6528bbe7-tmp\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.452634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452418    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-bin\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.452634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452474    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59qz\" (UniqueName: \"kubernetes.io/projected/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-kube-api-access-b59qz\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.452634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452523    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.452634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452554    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.452634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452586    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.452634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452616    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46099da2-dc06-45b7-bc3c-139b84c896e0-serviceca\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzlw\" (UniqueName: \"kubernetes.io/projected/5d1d8529-d285-41d3-b179-6f3343059567-kube-api-access-bmzlw\") pod \"volume-data-source-validator-58c9f5f46d-7mg8k\" (UID: \"5d1d8529-d285-41d3-b179-6f3343059567\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452678    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwtx\" (UniqueName: \"kubernetes.io/projected/5339a2af-2f4a-4444-b146-dcb849fa7449-kube-api-access-hzwtx\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452700    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfjn\" (UniqueName: \"kubernetes.io/projected/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-kube-api-access-mrfjn\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452718    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-host\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-script-lib\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452760    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbt8b\" (UniqueName: \"kubernetes.io/projected/14ba76bf-37ab-43fd-a583-5a9343e75964-kube-api-access-tbt8b\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452775    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-multus-certs\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.452814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452812    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-node-log\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-lib-modules\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452846    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-kubelet-config\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452893    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-ovn\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.452960    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5qs\" (UniqueName: \"kubernetes.io/projected/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-api-access-mr5qs\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-modprobe-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453051    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzgg\" (UniqueName: \"kubernetes.io/projected/22a93e48-3382-47af-adef-2ceb0d1fc6aa-kube-api-access-5wzgg\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453110    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-daemon-config\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.453163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453136    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvld\" (UniqueName: \"kubernetes.io/projected/23360850-74e3-473e-a579-70d4fcc5a9d0-kube-api-access-6gvld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453194    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-systemd-units\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-service-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453270    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-log-socket\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453297    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-bin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzcb\" (UniqueName: \"kubernetes.io/projected/cec9279a-2cd4-49bd-823c-398f6528bbe7-kube-api-access-whzcb\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453380    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrh4\" (UniqueName: \"kubernetes.io/projected/c5013b2f-c423-4c36-aaa0-917c35cf23f8-kube-api-access-hrrh4\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453480    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453525    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.453550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453557    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-run\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23360850-74e3-473e-a579-70d4fcc5a9d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453608    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-systemd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453655    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovn-node-metrics-cert\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysconfig\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453795    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec657dda-01bf-42d4-96de-1f9bb920dfff-agent-certs\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453908    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-hostroot\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453949    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovn-node-metrics-cert\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.453958    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-conf\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.454005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454003    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454092    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec657dda-01bf-42d4-96de-1f9bb920dfff-agent-certs\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454093    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dh4\" (UniqueName: \"kubernetes.io/projected/a63e5bbb-4666-48a9-8530-64848d5bf70b-kube-api-access-q7dh4\") pod \"migrator-66df5d496b-v4nfq\" (UID: \"a63e5bbb-4666-48a9-8530-64848d5bf70b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454128    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454195    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-cni-binary-copy\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454226    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454253    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-conf-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454283    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454311    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5815b6d3-0bdd-45b2-a673-5f493146c0f8-tmp-dir\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454339    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5n6\" (UniqueName: \"kubernetes.io/projected/0f4370e6-112e-4d65-940f-5ca5263ce913-kube-api-access-gg5n6\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454396    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da62aabf-4741-4ada-b763-f5d62d142ae4-crio-socket\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvg6b\" (UniqueName: \"kubernetes.io/projected/1f5f4127-5c51-4029-8ba2-0f5c01bad573-kube-api-access-lvg6b\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454477    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5815b6d3-0bdd-45b2-a673-5f493146c0f8-tmp-dir\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454480    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454505    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63257d8b-a976-4db1-8721-98f7d7fefed8-original-pull-secret\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.454521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454515    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-config\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454551    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454568    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhn7\" (UniqueName: \"kubernetes.io/projected/91cfe548-7bc2-4679-99a3-7212bdc92193-kube-api-access-zdhn7\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-etc-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454618    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-netd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-config\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454684    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.454750    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cec9279a-2cd4-49bd-823c-398f6528bbe7-serving-cert\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.455040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455033    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455050    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cec9279a-2cd4-49bd-823c-398f6528bbe7-serving-cert\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455057    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455081    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-netns\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455127    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-netns\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455147    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455165    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455203    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrt9\" (UniqueName: \"kubernetes.io/projected/5815b6d3-0bdd-45b2-a673-5f493146c0f8-kube-api-access-jbrt9\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bzx\" (UniqueName: \"kubernetes.io/projected/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-kube-api-access-t9bzx\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455245    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62l8\" (UniqueName: \"kubernetes.io/projected/a4ebbfa3-4272-4095-8a57-a55146be5135-kube-api-access-s62l8\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455266    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455283    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-config-volume\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455302    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jdn\" (UniqueName: \"kubernetes.io/projected/46099da2-dc06-45b7-bc3c-139b84c896e0-kube-api-access-v7jdn\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.455349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-system-cni-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455394    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455424    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5815b6d3-0bdd-45b2-a673-5f493146c0f8-hosts-file\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455474    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4ebbfa3-4272-4095-8a57-a55146be5135-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455521    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbfd\" (UniqueName: \"kubernetes.io/projected/b070df60-6cdb-43fa-842b-ca11d66b1770-kube-api-access-hpbfd\") pod \"migrator-66df5d496b-brq9g\" (UID: \"b070df60-6cdb-43fa-842b-ca11d66b1770\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455560    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-dbus\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455583    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455604    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da62aabf-4741-4ada-b763-f5d62d142ae4-data-volume\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455635    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-trusted-ca\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455653    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-env-overrides\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455684    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-multus\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.455848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.455698    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da62aabf-4741-4ada-b763-f5d62d142ae4-data-volume\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.458077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458061    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458283    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458295    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458447    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 11:29:57.458581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458507    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.458581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458508    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.458581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458577    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458610    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 11:29:57.458707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458626    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.458707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458640    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 11:29:57.458707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458660    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458755    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458808    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458824    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458842    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458856    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458864    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.458872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458864    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458883    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.458929    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459080    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459093    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459106    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459122    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459105    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459136    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459142    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459070    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459094    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459147    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459163    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459181    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459253    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459262    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.459526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459266    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459540    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459608    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459699    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459716    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459738    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459773    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459803    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 11:29:57.459847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459775    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 11:29:57.460116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.459993    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 11:29:57.460116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.460006    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:57.460116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.460083    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 11:29:57.460219 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.460206    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 11:29:57.462514 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.462497    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec657dda-01bf-42d4-96de-1f9bb920dfff-konnectivity-ca\") pod \"konnectivity-agent-58qds\" (UID: \"ec657dda-01bf-42d4-96de-1f9bb920dfff\") " pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:29:57.462599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.462525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-iptables-alerter-script\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.462599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.462577    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.462669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.462610    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c173a778-6503-4ebc-9bd4-750a98b0f9d9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:57.463214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.463196    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46099da2-dc06-45b7-bc3c-139b84c896e0-serviceca\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.463214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.463205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-script-lib\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.463390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.463377    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-daemon-config\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.463854 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.463840    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-service-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.463976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.463964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23360850-74e3-473e-a579-70d4fcc5a9d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:57.464324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.464307    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.464516 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.464498    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5339a2af-2f4a-4444-b146-dcb849fa7449-cni-binary-copy\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.464687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.464673    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.464847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.464834    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-config\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.464935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.464920    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-ovnkube-config\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.465488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.465471    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.465551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.465473    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.465551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.465519    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.465651 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.465565    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-config-volume\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.465733 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.465716    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f4370e6-112e-4d65-940f-5ca5263ce913-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.465910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.465896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fddeddb5-22c9-4d0b-a77f-6d2a12319856-env-overrides\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.471232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.471208    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 11:29:57.473005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.472986    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 11:29:57.473694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.473673    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 11:29:57.473743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.473727    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 11:29:57.473906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.473892    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.475620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.475604    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.475846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.475827    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4ebbfa3-4272-4095-8a57-a55146be5135-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.475972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.475955    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-trusted-ca\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.477476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.477440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297ff\" (UniqueName: \"kubernetes.io/projected/e3746ecb-830d-4d86-8400-21f4d2985b22-kube-api-access-297ff\") pod \"kas-connection-checker-685895c7c9-5b47m\" (UID: \"e3746ecb-830d-4d86-8400-21f4d2985b22\") " pod="kube-system/kas-connection-checker-685895c7c9-5b47m"
Oct 30 11:29:57.479842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.479823    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5wd\" (UniqueName: \"kubernetes.io/projected/fddeddb5-22c9-4d0b-a77f-6d2a12319856-kube-api-access-nj5wd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.479974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.479957    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm8x\" (UniqueName: \"kubernetes.io/projected/ea88fbd7-988d-44a5-a12e-fd5e64e8da79-kube-api-access-kpm8x\") pod \"kas-connection-checker-685895c7c9-x2vn2\" (UID: \"ea88fbd7-988d-44a5-a12e-fd5e64e8da79\") " pod="kube-system/kas-connection-checker-685895c7c9-x2vn2"
Oct 30 11:29:57.480974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.480953    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4mfs\" (UniqueName: \"kubernetes.io/projected/52a6b071-d9c7-4d71-aeae-65e818f74dec-kube-api-access-q4mfs\") pod \"network-check-source-6f78d59f99-7jfl7\" (UID: \"52a6b071-d9c7-4d71-aeae-65e818f74dec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jfl7"
Oct 30 11:29:57.481111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.481090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.481224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.481209    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") pod \"periodic-gathering-t6jts-b62vt\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") " pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:29:57.482109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.482089    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.482852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.482833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5b8\" (UniqueName: \"kubernetes.io/projected/8c68e591-1f33-4482-9843-d0412cec6bce-kube-api-access-nr5b8\") pod \"kas-connection-checker-685895c7c9-488kk\" (UID: \"8c68e591-1f33-4482-9843-d0412cec6bce\") " pod="kube-system/kas-connection-checker-685895c7c9-488kk"
Oct 30 11:29:57.483010 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.482991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec9279a-2cd4-49bd-823c-398f6528bbe7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.483069 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.483009    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc72p\" (UniqueName: \"kubernetes.io/projected/e40c5726-14a1-4f26-8d5f-218b4ff946a8-kube-api-access-lc72p\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.486259 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.486232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwtx\" (UniqueName: \"kubernetes.io/projected/5339a2af-2f4a-4444-b146-dcb849fa7449-kube-api-access-hzwtx\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.486372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.486297    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbt8b\" (UniqueName: \"kubernetes.io/projected/14ba76bf-37ab-43fd-a583-5a9343e75964-kube-api-access-tbt8b\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.486372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.486339    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcb6\" (UniqueName: \"kubernetes.io/projected/5c714f27-241c-4c7f-86fe-2c3eb89d0542-kube-api-access-bjcb6\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.486568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.486546    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvg6b\" (UniqueName: \"kubernetes.io/projected/1f5f4127-5c51-4029-8ba2-0f5c01bad573-kube-api-access-lvg6b\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.486613 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.486552    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrt9\" (UniqueName: \"kubernetes.io/projected/5815b6d3-0bdd-45b2-a673-5f493146c0f8-kube-api-access-jbrt9\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.487824 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.487806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.488481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.488446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bzx\" (UniqueName: \"kubernetes.io/projected/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-kube-api-access-t9bzx\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.491411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.491385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5n6\" (UniqueName: \"kubernetes.io/projected/0f4370e6-112e-4d65-940f-5ca5263ce913-kube-api-access-gg5n6\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.491623 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.491605    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.492048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.492029    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59qz\" (UniqueName: \"kubernetes.io/projected/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-kube-api-access-b59qz\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.492225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.492211    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvld\" (UniqueName: \"kubernetes.io/projected/23360850-74e3-473e-a579-70d4fcc5a9d0-kube-api-access-6gvld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz\" (UID: \"23360850-74e3-473e-a579-70d4fcc5a9d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz"
Oct 30 11:29:57.496956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.496926    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dh4\" (UniqueName: \"kubernetes.io/projected/a63e5bbb-4666-48a9-8530-64848d5bf70b-kube-api-access-q7dh4\") pod \"migrator-66df5d496b-v4nfq\" (UID: \"a63e5bbb-4666-48a9-8530-64848d5bf70b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v4nfq"
Oct 30 11:29:57.503420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.503400    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbfg\" (UniqueName: \"kubernetes.io/projected/cf8a23a0-a643-42bd-84f7-29a2bb9138b7-kube-api-access-mpbfg\") pod \"network-check-target-vxtn4\" (UID: \"cf8a23a0-a643-42bd-84f7-29a2bb9138b7\") " pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:29:57.503658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.503641    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhn7\" (UniqueName: \"kubernetes.io/projected/91cfe548-7bc2-4679-99a3-7212bdc92193-kube-api-access-zdhn7\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.506124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.506107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.506509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.506493    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbfd\" (UniqueName: \"kubernetes.io/projected/b070df60-6cdb-43fa-842b-ca11d66b1770-kube-api-access-hpbfd\") pod \"migrator-66df5d496b-brq9g\" (UID: \"b070df60-6cdb-43fa-842b-ca11d66b1770\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-brq9g"
Oct 30 11:29:57.506618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.506603    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzlw\" (UniqueName: \"kubernetes.io/projected/5d1d8529-d285-41d3-b179-6f3343059567-kube-api-access-bmzlw\") pod \"volume-data-source-validator-58c9f5f46d-7mg8k\" (UID: \"5d1d8529-d285-41d3-b179-6f3343059567\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7mg8k"
Oct 30 11:29:57.506714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.506697    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzgg\" (UniqueName: \"kubernetes.io/projected/22a93e48-3382-47af-adef-2ceb0d1fc6aa-kube-api-access-5wzgg\") pod \"ingress-canary-pnclc\" (UID: \"22a93e48-3382-47af-adef-2ceb0d1fc6aa\") " pod="openshift-ingress-canary/ingress-canary-pnclc"
Oct 30 11:29:57.507506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.507490    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfjn\" (UniqueName: \"kubernetes.io/projected/c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18-kube-api-access-mrfjn\") pod \"console-operator-7cd56b8dbc-scqxd\" (UID: \"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:29:57.508219 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.508199    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dhfxh_46099da2-dc06-45b7-bc3c-139b84c896e0/node-ca/0.log"
Oct 30 11:29:57.508802 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.508786    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jdn\" (UniqueName: \"kubernetes.io/projected/46099da2-dc06-45b7-bc3c-139b84c896e0-kube-api-access-v7jdn\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.509159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.509141    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62l8\" (UniqueName: \"kubernetes.io/projected/a4ebbfa3-4272-4095-8a57-a55146be5135-kube-api-access-s62l8\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.509649 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.509634    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrh4\" (UniqueName: \"kubernetes.io/projected/c5013b2f-c423-4c36-aaa0-917c35cf23f8-kube-api-access-hrrh4\") pod \"service-ca-operator-544d749c8c-wvjrq\" (UID: \"c5013b2f-c423-4c36-aaa0-917c35cf23f8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq"
Oct 30 11:29:57.511905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.511889    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5qs\" (UniqueName: \"kubernetes.io/projected/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-api-access-mr5qs\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.512300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.512281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzcb\" (UniqueName: \"kubernetes.io/projected/cec9279a-2cd4-49bd-823c-398f6528bbe7-kube-api-access-whzcb\") pod \"insights-operator-7c4986d449-25xnv\" (UID: \"cec9279a-2cd4-49bd-823c-398f6528bbe7\") " pod="openshift-insights/insights-operator-7c4986d449-25xnv"
Oct 30 11:29:57.556136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-systemd\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556084    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-sys\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556104    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-sys-fs\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.556136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556121    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.556136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556136    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-device-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556153    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-etc-kubernetes\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556178    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-var-lib-kubelet\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556198    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46099da2-dc06-45b7-bc3c-139b84c896e0-host\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556198    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-sys\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556198    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-systemd\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-socket-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556224    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-device-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556227    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-etc-kubernetes\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556250    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-var-lib-kubelet\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556267    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-host-slash\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-sys-fs\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556279    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-socket-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556291    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-kubelet\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556316    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-kubernetes\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556323    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-kubelet\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556279    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46099da2-dc06-45b7-bc3c-139b84c896e0-host\") pod \"node-ca-dhfxh\" (UID: \"46099da2-dc06-45b7-bc3c-139b84c896e0\") " pod="openshift-image-registry/node-ca-dhfxh"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556343    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-kubelet\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556315    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225-host-slash\") pod \"iptables-alerter-vhm89\" (UID: \"d1af9bdc-dd7e-4b70-9fb0-6a0a17b58225\") " pod="openshift-network-operator/iptables-alerter-vhm89"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556370    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-slash\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-cnibin\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556404    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-kubelet\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556409    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-slash\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556395    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-kubernetes\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556417    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556473    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556426    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-cnibin\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556494    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556519    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.556533    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556546    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-var-lib-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.556551 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.556564    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556578    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.556630    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls podName:1d000fdc-e5ed-4fce-bb1e-6715932a37b1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.056597088 +0000 UTC m=+1.851481575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls") pod "dns-default-zgz8g" (UID: "1d000fdc-e5ed-4fce-bb1e-6715932a37b1") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-var-lib-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556650    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-registration-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.556669    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.056658603 +0000 UTC m=+1.851543082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556699    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-socket-dir-parent\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556705    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-registration-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556717    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-k8s-cni-cncf-io\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556735    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-system-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-bin\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556760    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-socket-dir-parent\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556770    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556772    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-k8s-cni-cncf-io\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556787    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556798    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-system-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556801    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-bin\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556822    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-host\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556843    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-multus-certs\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556845    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-node-log\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556861    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-host\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556875    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-lib-modules\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556886    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-multus-certs\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556887    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c714f27-241c-4c7f-86fe-2c3eb89d0542-service-ca-bundle\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556897    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-kubelet-config\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-kubelet-config\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556901    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-node-log\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556945    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-ovn\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556975    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-modprobe-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-systemd-units\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.556981    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-ovn\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557027    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-log-socket\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-lib-modules\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557043    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557065    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-bin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557071    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-systemd-units\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-log-socket\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557087    5060 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557114    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-bin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557092    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557131    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-modprobe-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557154    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-run\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557181    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-systemd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557205    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysconfig\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557217    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-run\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557230    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-run-systemd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557257    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysconfig\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557282    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557304    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-hostroot\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.557318    5060 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91cfe548-7bc2-4679-99a3-7212bdc92193-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcf5w\" (UID: \"91cfe548-7bc2-4679-99a3-7212bdc92193\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcf5w"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557327    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-conf\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.557373    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.0573571 +0000 UTC m=+1.852241599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-conf-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557420    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557430    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-conf-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557394    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-hostroot\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557477    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da62aabf-4741-4ada-b763-f5d62d142ae4-crio-socket\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557485    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-conf\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557516    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-etc-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557539    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da62aabf-4741-4ada-b763-f5d62d142ae4-crio-socket\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-netd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557574    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-cni-netd\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557608    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-etc-openvswitch\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.557638    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.557738    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.557789    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.057773731 +0000 UTC m=+1.852658216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:57.557874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:57.557826    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:58.057816021 +0000 UTC m=+1.852700523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557926    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-netns\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557952    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-netns\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.557980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558006    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-system-cni-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558031    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5815b6d3-0bdd-45b2-a673-5f493146c0f8-hosts-file\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-dbus\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558106    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da62aabf-4741-4ada-b763-f5d62d142ae4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-multus\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558157    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558171    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-system-cni-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-cnibin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558210    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-netns\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-run-netns\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558280    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-cnibin\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558338    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5815b6d3-0bdd-45b2-a673-5f493146c0f8-hosts-file\") pod \"node-resolver-7lsq8\" (UID: \"5815b6d3-0bdd-45b2-a673-5f493146c0f8\") " pod="openshift-dns/node-resolver-7lsq8"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/14ba76bf-37ab-43fd-a583-5a9343e75964-etc-sysctl-d\") pod \"tuned-64dlw\" (UID: \"14ba76bf-37ab-43fd-a583-5a9343e75964\") " pod="openshift-cluster-node-tuning-operator/tuned-64dlw"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63257d8b-a976-4db1-8721-98f7d7fefed8-dbus\") pod \"global-pull-secret-syncer-r4xxg\" (UID: \"63257d8b-a976-4db1-8721-98f7d7fefed8\") " pod="kube-system/global-pull-secret-syncer-r4xxg"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558429    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fddeddb5-22c9-4d0b-a77f-6d2a12319856-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2zn7\" (UID: \"fddeddb5-22c9-4d0b-a77f-6d2a12319856\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-multus-cni-dir\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558514    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5339a2af-2f4a-4444-b146-dcb849fa7449-host-var-lib-cni-multus\") pod \"multus-289w4\" (UID: \"5339a2af-2f4a-4444-b146-dcb849fa7449\") " pod="openshift-multus/multus-289w4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.558525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f4370e6-112e-4d65-940f-5ca5263ce913-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2ml5\" (UID: \"0f4370e6-112e-4d65-940f-5ca5263ce913\") " pod="openshift-multus/multus-additional-cni-plugins-d2ml5"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.559611    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"image-registry-7d654c48b4-zcdvc\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") " pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.559839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f5f4127-5c51-4029-8ba2-0f5c01bad573-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lwp9b\" (UID: \"1f5f4127-5c51-4029-8ba2-0f5c01bad573\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.560085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e40c5726-14a1-4f26-8d5f-218b4ff946a8-metrics-certs\") pod \"network-metrics-daemon-6s7j4\" (UID: \"e40c5726-14a1-4f26-8d5f-218b4ff946a8\") " pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.560199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.560153    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"image-registry-7f7dc85469-gjnw7\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") " pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.593292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.593256    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:57.638573 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.638546    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"
Oct 30 11:29:57.639598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.639582    5060 scope.go:122] "RemoveContainer" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572"
Oct 30 11:29:57.639910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.639886    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:29:57.645082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.645060    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s7j4"
Oct 30 11:29:57.649874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.649856    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:57.762723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.762122    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:57.962182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.962152    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:57.966633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:57.966607    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b"]
Oct 30 11:29:58.017693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.017672    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f7dc85469-gjnw7"]
Oct 30 11:29:58.018719 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:58.018686    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58c2f33_be41_4e16_91bc_293ee27d908e.slice/crio-47e08c4f88d9fb0d1146becd6cf6365c28f891594144540db67afb52cf2a82d7 WatchSource:0}: Error finding container 47e08c4f88d9fb0d1146becd6cf6365c28f891594144540db67afb52cf2a82d7: Status 404 returned error can't find the container with id 47e08c4f88d9fb0d1146becd6cf6365c28f891594144540db67afb52cf2a82d7
Oct 30 11:29:58.036299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.036080    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d654c48b4-zcdvc"]
Oct 30 11:29:58.037994 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:58.037967    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebf069a_0c3e_44c6_9ad8_23dc0056c442.slice/crio-647257c75a2e0ced4b122d62f35ebe5ed4016091cd76525e390f59b942dc5b4b WatchSource:0}: Error finding container 647257c75a2e0ced4b122d62f35ebe5ed4016091cd76525e390f59b942dc5b4b: Status 404 returned error can't find the container with id 647257c75a2e0ced4b122d62f35ebe5ed4016091cd76525e390f59b942dc5b4b
Oct 30 11:29:58.040318 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.040297    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:29:58.061516 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.061488    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:58.061614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.061554    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:58.061614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.061587    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:58.061684 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061654    5060 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:58.061722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.061690    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:58.061771 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061728    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.061707762 +0000 UTC m=+2.856592257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:58.061771 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061738    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:58.061771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.061754    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:58.061906 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061808    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.061790884 +0000 UTC m=+2.856675375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:58.061906 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061847    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:58.061906 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061885    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.061872684 +0000 UTC m=+2.856757178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:58.062066 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.061983    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:58.062066 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.062050    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.062037274 +0000 UTC m=+2.856921756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:58.064223 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.064200    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d000fdc-e5ed-4fce-bb1e-6715932a37b1-metrics-tls\") pod \"dns-default-zgz8g\" (UID: \"1d000fdc-e5ed-4fce-bb1e-6715932a37b1\") " pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:58.068568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.068550    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6s7j4"]
Oct 30 11:29:58.070314 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:58.070292    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40c5726_14a1_4f26_8d5f_218b4ff946a8.slice/crio-e3454827bc96d67cab305298526c96271ea65b91d5e7afbd8a7a64a8b4c1236a WatchSource:0}: Error finding container e3454827bc96d67cab305298526c96271ea65b91d5e7afbd8a7a64a8b4c1236a: Status 404 returned error can't find the container with id e3454827bc96d67cab305298526c96271ea65b91d5e7afbd8a7a64a8b4c1236a
Oct 30 11:29:58.095373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.095344    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pnclc_22a93e48-3382-47af-adef-2ceb0d1fc6aa/serve-healthcheck-canary/0.log"
Oct 30 11:29:58.170850 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.170812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:58.250708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.250678    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:29:58.295235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.295209    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-brq9g_b070df60-6cdb-43fa-842b-ca11d66b1770/migrator/0.log"
Oct 30 11:29:58.313047 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.313017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:58.394959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.394914    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s7j4" event={"ID":"e40c5726-14a1-4f26-8d5f-218b4ff946a8","Type":"ContainerStarted","Data":"e3454827bc96d67cab305298526c96271ea65b91d5e7afbd8a7a64a8b4c1236a"}
Oct 30 11:29:58.396628 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.396602    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:29:58.397091 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.397072    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/1.log"
Oct 30 11:29:58.397157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.397115    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" containerID="91a85978695cae0a5cd7c5a01a551846d66211febb517c2786a802e81171aed2" exitCode=255
Oct 30 11:29:58.397218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.397194    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" event={"ID":"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18","Type":"ContainerDied","Data":"91a85978695cae0a5cd7c5a01a551846d66211febb517c2786a802e81171aed2"}
Oct 30 11:29:58.397266 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.397255    5060 scope.go:122] "RemoveContainer" containerID="c0eec9d021dde0a9339564d95341ff7ccdd4e0423be25bb8ba51ef967bea4572"
Oct 30 11:29:58.397813 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.397622    5060 scope.go:122] "RemoveContainer" containerID="91a85978695cae0a5cd7c5a01a551846d66211febb517c2786a802e81171aed2"
Oct 30 11:29:58.397897 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:58.397874    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scqxd_openshift-console-operator(c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18"
Oct 30 11:29:58.399640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.399605    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc" event={"ID":"bebf069a-0c3e-44c6-9ad8-23dc0056c442","Type":"ContainerStarted","Data":"f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c"}
Oct 30 11:29:58.399741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.399647    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc" event={"ID":"bebf069a-0c3e-44c6-9ad8-23dc0056c442","Type":"ContainerStarted","Data":"647257c75a2e0ced4b122d62f35ebe5ed4016091cd76525e390f59b942dc5b4b"}
Oct 30 11:29:58.399741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.399733    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:29:58.401227 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.401206    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7" event={"ID":"d58c2f33-be41-4e16-91bc-293ee27d908e","Type":"ContainerStarted","Data":"a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5"}
Oct 30 11:29:58.401321 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.401259    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7" event={"ID":"d58c2f33-be41-4e16-91bc-293ee27d908e","Type":"ContainerStarted","Data":"47e08c4f88d9fb0d1146becd6cf6365c28f891594144540db67afb52cf2a82d7"}
Oct 30 11:29:58.402375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.402352    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b" event={"ID":"1f5f4127-5c51-4029-8ba2-0f5c01bad573","Type":"ContainerStarted","Data":"9e68eef658c9eb1162f31ea27e3c4d05b79801b7cbe062ae57d9fd0496102f48"}
Oct 30 11:29:58.457015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.456972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:58.494808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.494779    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-v4nfq_a63e5bbb-4666-48a9-8530-64848d5bf70b/migrator/0.log"
Oct 30 11:29:58.569900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.569876    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgz8g"]
Oct 30 11:29:58.571540 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:29:58.571509    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d000fdc_e5ed_4fce_bb1e_6715932a37b1.slice/crio-cfa5d8058b187e213f1eb112c1431cc9e2b18cb4a3965cf04fd74befab635edb WatchSource:0}: Error finding container cfa5d8058b187e213f1eb112c1431cc9e2b18cb4a3965cf04fd74befab635edb: Status 404 returned error can't find the container with id cfa5d8058b187e213f1eb112c1431cc9e2b18cb4a3965cf04fd74befab635edb
Oct 30 11:29:58.604230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.604197    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:58.609383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.609356    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:29:58.695404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.695368    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz_23360850-74e3-473e-a579-70d4fcc5a9d0/kube-storage-version-migrator-operator/0.log"
Oct 30 11:29:58.751681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:58.751603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:59.072470 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.072408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:29:59.072648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.072520    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:29:59.072648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.072580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:29:59.072648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.072612    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:29:59.072823 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.072745    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:59.072823 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.072815    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:01.0728005 +0000 UTC m=+4.867684976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:59.073210 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.073191    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:29:59.073314 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.073236    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:01.07322527 +0000 UTC m=+4.868109742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:29:59.073314 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.073291    5060 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:29:59.073425 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.073323    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:01.073313107 +0000 UTC m=+4.868197583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:29:59.073425 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.073370    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:59.073425 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:29:59.073400    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:01.073391106 +0000 UTC m=+4.868275583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:29:59.411309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.411273    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s7j4" event={"ID":"e40c5726-14a1-4f26-8d5f-218b4ff946a8","Type":"ContainerStarted","Data":"0877042f3e97e0d82eb232d208752bb8effdfd8720c897a264e1593526d5a1ba"}
Oct 30 11:29:59.413360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.413336    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:29:59.414810 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.414723    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgz8g" event={"ID":"1d000fdc-e5ed-4fce-bb1e-6715932a37b1","Type":"ContainerStarted","Data":"cfa5d8058b187e213f1eb112c1431cc9e2b18cb4a3965cf04fd74befab635edb"}
Oct 30 11:29:59.415499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.415038    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:29:59.942039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:29:59.941986    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:00.139283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.139255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:00.354114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.354086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:00.419441 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.419415    5060 generic.go:356] "Generic (PLEG): container finished" podID="cec9279a-2cd4-49bd-823c-398f6528bbe7" containerID="b0c4044544a5156369b74978b964fe1251af2e030f7a118a58f537d40701cbb6" exitCode=0
Oct 30 11:30:00.419887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.419499    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-25xnv" event={"ID":"cec9279a-2cd4-49bd-823c-398f6528bbe7","Type":"ContainerDied","Data":"b0c4044544a5156369b74978b964fe1251af2e030f7a118a58f537d40701cbb6"}
Oct 30 11:30:00.419950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.419889    5060 scope.go:122] "RemoveContainer" containerID="b0c4044544a5156369b74978b964fe1251af2e030f7a118a58f537d40701cbb6"
Oct 30 11:30:00.421307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.421244    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b" event={"ID":"1f5f4127-5c51-4029-8ba2-0f5c01bad573","Type":"ContainerStarted","Data":"38e46b0fce6a4a54837a43041aabd9227daf7526070350bcb88cdb2ba0bc458e"}
Oct 30 11:30:00.421307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.421287    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lwp9b" event={"ID":"1f5f4127-5c51-4029-8ba2-0f5c01bad573","Type":"ContainerStarted","Data":"1c1618379c1ed711dfa1e394fddca5307f72459dcfa6a1af753ac6b4ed208c89"}
Oct 30 11:30:00.423180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.423154    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s7j4" event={"ID":"e40c5726-14a1-4f26-8d5f-218b4ff946a8","Type":"ContainerStarted","Data":"2b6a170a2df18373c3742d2d2007fad110448f47cda9bf5b55a2671b38d35c88"}
Oct 30 11:30:00.424634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.424614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgz8g" event={"ID":"1d000fdc-e5ed-4fce-bb1e-6715932a37b1","Type":"ContainerStarted","Data":"72d0902080f1f921d26b967909eeaec47adf87a77e7cca23fbad02d57078756b"}
Oct 30 11:30:00.538151 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.538120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:00.734619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:00.734587    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.094330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.094395    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.094501    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.094576    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.094736    5060 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.094818    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert podName:c173a778-6503-4ebc-9bd4-750a98b0f9d9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:05.09479138 +0000 UTC m=+8.889675866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-b5tmq" (UID: "c173a778-6503-4ebc-9bd4-750a98b0f9d9") : secret "networking-console-plugin-cert" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.095294    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.095353    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls podName:a4ebbfa3-4272-4095-8a57-a55146be5135 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:05.095331334 +0000 UTC m=+8.890215808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nwx29" (UID: "a4ebbfa3-4272-4095-8a57-a55146be5135") : secret "cluster-monitoring-operator-tls" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.095422    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.095483    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:05.095472249 +0000 UTC m=+8.890356734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.095553    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 11:30:01.096536 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.095591    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs podName:5c714f27-241c-4c7f-86fe-2c3eb89d0542 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:05.095580235 +0000 UTC m=+8.890464709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs") pod "router-default-7d8f76c5df-xxgbt" (UID: "5c714f27-241c-4c7f-86fe-2c3eb89d0542") : secret "router-metrics-certs-default" not found
Oct 30 11:30:01.256171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.256141    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:30:01.256551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.256536    5060 scope.go:122] "RemoveContainer" containerID="91a85978695cae0a5cd7c5a01a551846d66211febb517c2786a802e81171aed2"
Oct 30 11:30:01.256756 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:01.256737    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scqxd_openshift-console-operator(c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18"
Oct 30 11:30:01.430127 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.430040    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgz8g" event={"ID":"1d000fdc-e5ed-4fce-bb1e-6715932a37b1","Type":"ContainerStarted","Data":"1c1e187e671d09aaebdfd9f1226185f49ae950e2939207d9386069bae0bd5af2"}
Oct 30 11:30:01.430555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.430170    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:30:01.431528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.431509    5060 generic.go:356] "Generic (PLEG): container finished" podID="23360850-74e3-473e-a579-70d4fcc5a9d0" containerID="3f249043c4e3ab6c8f4668aed03cf189b0495fe6309f2e2177cd02546f1e20be" exitCode=0
Oct 30 11:30:01.431679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.431577    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz" event={"ID":"23360850-74e3-473e-a579-70d4fcc5a9d0","Type":"ContainerDied","Data":"3f249043c4e3ab6c8f4668aed03cf189b0495fe6309f2e2177cd02546f1e20be"}
Oct 30 11:30:01.431845 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.431829    5060 scope.go:122] "RemoveContainer" containerID="3f249043c4e3ab6c8f4668aed03cf189b0495fe6309f2e2177cd02546f1e20be"
Oct 30 11:30:01.433396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.433362    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-25xnv" event={"ID":"cec9279a-2cd4-49bd-823c-398f6528bbe7","Type":"ContainerStarted","Data":"9162e95c8a256634dc92b4bcc8a11b6819d91518a1cbd5918c272f9379d751f8"}
Oct 30 11:30:01.965741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:01.965712    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:02.138803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:02.138770    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:02.339996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:02.339910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:02.439423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:02.439387    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kvvfz" event={"ID":"23360850-74e3-473e-a579-70d4fcc5a9d0","Type":"ContainerStarted","Data":"9cda5e1e3416ed2d4aaa06b7a8d044f43fba7d388bbb87a94889ad15d1ed0a9c"}
Oct 30 11:30:02.538517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:02.538479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:02.735976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:02.735944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:03.939532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:03.939492    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:04.140826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:04.140796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:04.341354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:04.341322    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:04.510041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:04.510010    5060 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:30:04.510420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:04.510392    5060 scope.go:122] "RemoveContainer" containerID="91a85978695cae0a5cd7c5a01a551846d66211febb517c2786a802e81171aed2"
Oct 30 11:30:04.510632 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:04.510611    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scqxd_openshift-console-operator(c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18"
Oct 30 11:30:04.539993 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:04.539939    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:04.766844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:04.766812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:05.129696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.129603    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:05.129696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.129690    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:30:05.130166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.129732    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:30:05.130166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.129759    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:30:05.130166 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:05.129862    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:05.130166 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:05.129963    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls podName:da62aabf-4741-4ada-b763-f5d62d142ae4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:13.129943316 +0000 UTC m=+16.924827807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x9lhg" (UID: "da62aabf-4741-4ada-b763-f5d62d142ae4") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:05.132364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.132332    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c714f27-241c-4c7f-86fe-2c3eb89d0542-metrics-certs\") pod \"router-default-7d8f76c5df-xxgbt\" (UID: \"5c714f27-241c-4c7f-86fe-2c3eb89d0542\") " pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:05.132499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.132343    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c173a778-6503-4ebc-9bd4-750a98b0f9d9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-b5tmq\" (UID: \"c173a778-6503-4ebc-9bd4-750a98b0f9d9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:30:05.132499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.132433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ebbfa3-4272-4095-8a57-a55146be5135-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nwx29\" (UID: \"a4ebbfa3-4272-4095-8a57-a55146be5135\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:30:05.137198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.137179    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:05.137240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.137225    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"
Oct 30 11:30:05.141114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.141090    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"
Oct 30 11:30:05.467186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.467159    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-b5tmq"]
Oct 30 11:30:05.468653 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:30:05.468561    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc173a778_6503_4ebc_9bd4_750a98b0f9d9.slice/crio-5a2eb1d446345d8f86c97b94e92b82c4bc57a4121493df3578334627004de1a3 WatchSource:0}: Error finding container 5a2eb1d446345d8f86c97b94e92b82c4bc57a4121493df3578334627004de1a3: Status 404 returned error can't find the container with id 5a2eb1d446345d8f86c97b94e92b82c4bc57a4121493df3578334627004de1a3
Oct 30 11:30:05.471440 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:30:05.471396    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ebbfa3_4272_4095_8a57_a55146be5135.slice/crio-821a607892d1ad3a996898d1a76492135900091da5555d09e23bc091037ab56a WatchSource:0}: Error finding container 821a607892d1ad3a996898d1a76492135900091da5555d09e23bc091037ab56a: Status 404 returned error can't find the container with id 821a607892d1ad3a996898d1a76492135900091da5555d09e23bc091037ab56a
Oct 30 11:30:05.476094 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:30:05.476073    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c714f27_241c_4c7f_86fe_2c3eb89d0542.slice/crio-1212beaa7254d81fb6aabc3bc685e31e6e88df2cdb15dbc7460a194788ad47a6 WatchSource:0}: Error finding container 1212beaa7254d81fb6aabc3bc685e31e6e88df2cdb15dbc7460a194788ad47a6: Status 404 returned error can't find the container with id 1212beaa7254d81fb6aabc3bc685e31e6e88df2cdb15dbc7460a194788ad47a6
Oct 30 11:30:05.479588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.479567    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29"]
Oct 30 11:30:05.480389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.480368    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d8f76c5df-xxgbt"]
Oct 30 11:30:05.653506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.653469    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:05.717444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.717353    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:30:05.795098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:05.795067    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:06.138364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.138283    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:06.281616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.281576    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:06.431495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.431393    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:06.454891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.454860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt" event={"ID":"5c714f27-241c-4c7f-86fe-2c3eb89d0542","Type":"ContainerStarted","Data":"fd082ca2ba2b04ab0769e3a2b13867409823d1f2b445e67bbd1808547bc443a7"}
Oct 30 11:30:06.455079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.454913    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt" event={"ID":"5c714f27-241c-4c7f-86fe-2c3eb89d0542","Type":"ContainerStarted","Data":"1212beaa7254d81fb6aabc3bc685e31e6e88df2cdb15dbc7460a194788ad47a6"}
Oct 30 11:30:06.456320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.456290    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq" event={"ID":"c173a778-6503-4ebc-9bd4-750a98b0f9d9","Type":"ContainerStarted","Data":"5a2eb1d446345d8f86c97b94e92b82c4bc57a4121493df3578334627004de1a3"}
Oct 30 11:30:06.457873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.457852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29" event={"ID":"a4ebbfa3-4272-4095-8a57-a55146be5135","Type":"ContainerStarted","Data":"821a607892d1ad3a996898d1a76492135900091da5555d09e23bc091037ab56a"}
Oct 30 11:30:06.466809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.466782    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:30:06.467015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.467001    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:30:06.581989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.581947    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:06.740440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.740401    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:06.745979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.745956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:30:06.959939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:06.959910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:07.138411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:07.138373    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:07.140910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:07.140888    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:07.464256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:07.464220    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-b5tmq" event={"ID":"c173a778-6503-4ebc-9bd4-750a98b0f9d9","Type":"ContainerStarted","Data":"1bc2ae64bfb427e3000411ff1de95221dd7ffa5fbc01975761177b2d24aee0d3"}
Oct 30 11:30:07.465898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:07.465871    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nwx29" event={"ID":"a4ebbfa3-4272-4095-8a57-a55146be5135","Type":"ContainerStarted","Data":"42ffea99c05985b7c40f1ebd30769b17107918cf21daeef00a695f1a83663b66"}
Oct 30 11:30:07.622106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:07.622069    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:07.783311 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:07.783230    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:08.089160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:08.089072    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:08.235768 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:08.235729    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:08.469402 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:08.469373    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:30:09.100709 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.100680    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2zn7"
Oct 30 11:30:09.227586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.227552    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:09.228943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.228923    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d8f76c5df-xxgbt"
Oct 30 11:30:09.326900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.326872    5060 scope.go:122] "RemoveContainer" containerID="91a85978695cae0a5cd7c5a01a551846d66211febb517c2786a802e81171aed2"
Oct 30 11:30:09.474713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.474686    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:30:09.474867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.474799    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" event={"ID":"c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18","Type":"ContainerStarted","Data":"15f1156e3ae2b7d9344b5ac0aa9f7baa799007f293ba7afb161369a3638903a0"}
Oct 30 11:30:09.475182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.475160    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:30:09.476210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.476186    5060 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-scqxd container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Oct 30 11:30:09.476310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:09.476225    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd" podUID="c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Oct 30 11:30:10.027172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:10.027128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:10.196479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:10.196427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:10.347820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:10.347734    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:10.483483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:10.483435    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scqxd"
Oct 30 11:30:11.442272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:11.442244    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zgz8g"
Oct 30 11:30:12.439972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:12.439935    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:12.593907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:12.593870    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:12.826700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:12.826669    5060 patch_prober.go:29] interesting pod/image-registry-7d654c48b4-zcdvc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 11:30:12.826867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:12.826732    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc" podUID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:30:13.201436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.201398    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:30:13.204446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.204415    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da62aabf-4741-4ada-b763-f5d62d142ae4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x9lhg\" (UID: \"da62aabf-4741-4ada-b763-f5d62d142ae4\") " pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:30:13.251165 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.251132    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x9lhg"
Oct 30 11:30:13.577168 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:30:13.577131    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda62aabf_4741_4ada_b763_f5d62d142ae4.slice/crio-f2a095459b64786617435bfffa813ebf9c8983289915b665c513c81ce5786e66 WatchSource:0}: Error finding container f2a095459b64786617435bfffa813ebf9c8983289915b665c513c81ce5786e66: Status 404 returned error can't find the container with id f2a095459b64786617435bfffa813ebf9c8983289915b665c513c81ce5786e66
Oct 30 11:30:13.578990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.578858    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x9lhg"]
Oct 30 11:30:13.595524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.595492    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hb6nn-fxr7h"]
Oct 30 11:30:13.600435 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.600410    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.604640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.604615    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.604763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.604650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.604763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.604677    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.604886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.604863    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6s8\" (UniqueName: \"kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.637434 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.637395    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hb6nn-fxr7h"]
Oct 30 11:30:13.653238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.651953    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f7dc85469-gjnw7"]
Oct 30 11:30:13.657156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.657127    5060 patch_prober.go:29] interesting pod/image-registry-7f7dc85469-gjnw7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 11:30:13.657321 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.657186    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7" podUID="d58c2f33-be41-4e16-91bc-293ee27d908e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:30:13.695158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.695111    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:13.705874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.705834    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6s8\" (UniqueName: \"kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.706072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.705891    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.706072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.705921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.706072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.705957    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.706402 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.706378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.706588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.706570    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.707049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.707031    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.743494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.743438    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6s8\" (UniqueName: \"kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8\") pod \"periodic-gathering-hb6nn-fxr7h\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") " pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:13.910247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.910157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:13.926766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:13.926737    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:30:14.108428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.108388    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:14.230499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.230477    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hb6nn-fxr7h"]
Oct 30 11:30:14.232354 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:30:14.232324    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2822b498_a2ec_4914_9846_1ee191542e12.slice/crio-baa78990d3eee18b84c25d885347b0ca9d7d45408a1c8c8310eacf021067e7d3 WatchSource:0}: Error finding container baa78990d3eee18b84c25d885347b0ca9d7d45408a1c8c8310eacf021067e7d3: Status 404 returned error can't find the container with id baa78990d3eee18b84c25d885347b0ca9d7d45408a1c8c8310eacf021067e7d3
Oct 30 11:30:14.496525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.496422    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h" event={"ID":"2822b498-a2ec-4914-9846-1ee191542e12","Type":"ContainerStarted","Data":"3234fcce45cba4825ef444d1e7aec2d94a935ad5a0192431c94938a752c1d15b"}
Oct 30 11:30:14.496525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.496508    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h" event={"ID":"2822b498-a2ec-4914-9846-1ee191542e12","Type":"ContainerStarted","Data":"baa78990d3eee18b84c25d885347b0ca9d7d45408a1c8c8310eacf021067e7d3"}
Oct 30 11:30:14.498234 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.498213    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x9lhg" event={"ID":"da62aabf-4741-4ada-b763-f5d62d142ae4","Type":"ContainerStarted","Data":"b9dc988ecf0e0d393014da5384f9dba995a8455122ec0a5cbcf09ffe7fddf82e"}
Oct 30 11:30:14.498339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.498254    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x9lhg" event={"ID":"da62aabf-4741-4ada-b763-f5d62d142ae4","Type":"ContainerStarted","Data":"fb4b7e5d89facdf331adadc6cb605a383606dad5d5f30d48954dbdfaeb59c498"}
Oct 30 11:30:14.498339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.498271    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x9lhg" event={"ID":"da62aabf-4741-4ada-b763-f5d62d142ae4","Type":"ContainerStarted","Data":"f2a095459b64786617435bfffa813ebf9c8983289915b665c513c81ce5786e66"}
Oct 30 11:30:14.538755 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:14.538691    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h" podStartSLOduration=1.538673641 podStartE2EDuration="1.538673641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:14.53811226 +0000 UTC m=+18.332996755" watchObservedRunningTime="2026-10-30 11:30:14.538673641 +0000 UTC m=+18.333558135"
Oct 30 11:30:15.655488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:15.655442    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgz8g_1d000fdc-e5ed-4fce-bb1e-6715932a37b1/dns/0.log"
Oct 30 11:30:15.856289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:15.856217    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgz8g_1d000fdc-e5ed-4fce-bb1e-6715932a37b1/kube-rbac-proxy/0.log"
Oct 30 11:30:16.055763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:16.055738    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lsq8_5815b6d3-0bdd-45b2-a673-5f493146c0f8/dns-node-resolver/0.log"
Oct 30 11:30:16.509739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:16.509696    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x9lhg" event={"ID":"da62aabf-4741-4ada-b763-f5d62d142ae4","Type":"ContainerStarted","Data":"68e39097f2517b7d137bcf9a4155fc4ec030a362e85d6e3422f2d1321cfe58b7"}
Oct 30 11:30:16.856356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:16.856276    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d654c48b4-zcdvc_bebf069a-0c3e-44c6-9ad8-23dc0056c442/registry/0.log"
Oct 30 11:30:17.062225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:17.062191    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f7dc85469-gjnw7_d58c2f33-be41-4e16-91bc-293ee27d908e/registry/0.log"
Oct 30 11:30:17.654993 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:17.654970    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dhfxh_46099da2-dc06-45b7-bc3c-139b84c896e0/node-ca/0.log"
Oct 30 11:30:17.856410 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:17.856381    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8f76c5df-xxgbt_5c714f27-241c-4c7f-86fe-2c3eb89d0542/router/0.log"
Oct 30 11:30:18.215759 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:18.215726    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:18.255688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:18.255656    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pnclc_22a93e48-3382-47af-adef-2ceb0d1fc6aa/serve-healthcheck-canary/0.log"
Oct 30 11:30:18.378833 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:18.378800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:18.536367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:18.536291    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:19.419234 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:19.419206    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:30:19.855684 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:19.855646    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:30:19.857906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:19.857869    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:20.003711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:20.003685    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:20.306770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:20.306731    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:20.932973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:20.932927    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:21.278034 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:21.277986    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:21.452750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:21.452719    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:21.683954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:21.683876    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:21.837891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:21.837855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:22.707283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:22.707247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:22.892116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:22.892083    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:23.002296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.002225    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7d764"]
Oct 30 11:30:23.041036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.041013    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.050178 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.050144    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:23.062593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.062567    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 11:30:23.062693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.062686    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-frflv\""
Oct 30 11:30:23.075213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.075198    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 11:30:23.081732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.081713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-wtmp\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.081816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.081764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-textfile\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.081885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.081868    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.081927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.081895    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.081999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.081938    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-tls\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.081999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.081962    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-sys\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.082059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.082025    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e33e6c3-9095-4ac7-8df7-c97744686f2c-metrics-client-ca\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.082059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.082044    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xv5v\" (UniqueName: \"kubernetes.io/projected/4e33e6c3-9095-4ac7-8df7-c97744686f2c-kube-api-access-4xv5v\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.082117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.082066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-root\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.133961 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.133932    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 11:30:23.134074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.134021    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 11:30:23.182575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-wtmp\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182591    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-textfile\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182625    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182652    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-tls\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182710    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-sys\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182722    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-wtmp\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182740    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e33e6c3-9095-4ac7-8df7-c97744686f2c-metrics-client-ca\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182768    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xv5v\" (UniqueName: \"kubernetes.io/projected/4e33e6c3-9095-4ac7-8df7-c97744686f2c-kube-api-access-4xv5v\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182773    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-sys\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182817    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-root\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.182957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.182885    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e33e6c3-9095-4ac7-8df7-c97744686f2c-root\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.183245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.183189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-textfile\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.183367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.183349    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e33e6c3-9095-4ac7-8df7-c97744686f2c-metrics-client-ca\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.183437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.183355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.185092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.185073    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.185255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.185238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e33e6c3-9095-4ac7-8df7-c97744686f2c-node-exporter-tls\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.200197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.200171    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:23.206291 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.206269    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xv5v\" (UniqueName: \"kubernetes.io/projected/4e33e6c3-9095-4ac7-8df7-c97744686f2c-kube-api-access-4xv5v\") pod \"node-exporter-7d764\" (UID: \"4e33e6c3-9095-4ac7-8df7-c97744686f2c\") " pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.393937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.393858    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7d764"
Oct 30 11:30:23.403199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.403175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:23.423709 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:30:23.423676    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e33e6c3_9095_4ac7_8df7_c97744686f2c.slice/crio-285cc74618ebb792334184260303a335cb0025fbade52853613447d97ff9b101 WatchSource:0}: Error finding container 285cc74618ebb792334184260303a335cb0025fbade52853613447d97ff9b101: Status 404 returned error can't find the container with id 285cc74618ebb792334184260303a335cb0025fbade52853613447d97ff9b101
Oct 30 11:30:23.536842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.536803    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7d764" event={"ID":"4e33e6c3-9095-4ac7-8df7-c97744686f2c","Type":"ContainerStarted","Data":"285cc74618ebb792334184260303a335cb0025fbade52853613447d97ff9b101"}
Oct 30 11:30:23.562444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.562416    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:23.657029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:23.656976    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:30:24.202534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:24.202496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:24.363325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:24.363287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:24.530563 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:24.530491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:24.866818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:24.866790    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-58qds"
Oct 30 11:30:25.546061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:25.546030    5060 generic.go:356] "Generic (PLEG): container finished" podID="4e33e6c3-9095-4ac7-8df7-c97744686f2c" containerID="b8dd3ee47a5c6034a2573f4e1c85850323385703ee298edca1895b5de17bc87c" exitCode=0
Oct 30 11:30:25.546475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:25.546114    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7d764" event={"ID":"4e33e6c3-9095-4ac7-8df7-c97744686f2c","Type":"ContainerDied","Data":"b8dd3ee47a5c6034a2573f4e1c85850323385703ee298edca1895b5de17bc87c"}
Oct 30 11:30:25.720712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:25.720684    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vxtn4"
Oct 30 11:30:26.286782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:26.286741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:26.451719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:26.451666    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:26.552390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:26.552311    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7d764" event={"ID":"4e33e6c3-9095-4ac7-8df7-c97744686f2c","Type":"ContainerStarted","Data":"a09ae541d01a561159a07926df4f4e87da206838f2bb462123609bb6dd8cb198"}
Oct 30 11:30:26.552390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:26.552363    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7d764" event={"ID":"4e33e6c3-9095-4ac7-8df7-c97744686f2c","Type":"ContainerStarted","Data":"947c63b0db5f4002d37da39c67851ae7d7818ec9abdb03d1e5f50806221fe231"}
Oct 30 11:30:26.611630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:26.611585    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7d764" podStartSLOduration=3.322873687 podStartE2EDuration="4.611569601s" totalImagesPullingTime="1.200471988s" totalInitContainerRuntime="88.223926ms" isStatefulPod=true podCreationTimestamp="2026-10-30 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:26.610623748 +0000 UTC m=+30.405508259" watchObservedRunningTime="2026-10-30 11:30:26.611569601 +0000 UTC m=+30.406454090"
Oct 30 11:30:27.445353 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:27.445313    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:27.623952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:27.623642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:27.847603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:27.847570    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:28.047251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:28.047217    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:28.261390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:28.261355    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:28.420951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:28.420923    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:30.058586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:30.058531    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:30.256798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:30.256762    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:30.413423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:30.413339    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:34.093110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:34.093076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:34.240197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:34.240166    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:37.104541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:37.104442    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d654c48b4-zcdvc"]
Oct 30 11:30:37.278692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:37.278659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:37.438789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:37.438706    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:37.619052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:37.619007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:38.675931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:38.675891    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7" podUID="d58c2f33-be41-4e16-91bc-293ee27d908e" containerName="registry" containerID="cri-o://a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5" gracePeriod=30
Oct 30 11:30:38.914328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:38.914304    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:30:38.951304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:38.951274    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:39.030837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.030802    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.030851    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.030891    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.030917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.030943    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.031062    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.031126    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.031160    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") pod \"d58c2f33-be41-4e16-91bc-293ee27d908e\" (UID: \"d58c2f33-be41-4e16-91bc-293ee27d908e\") "
Oct 30 11:30:39.031397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.031315    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:30:39.031570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.031489    5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-certificates\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.031882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.031811    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:30:39.033671 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.033645    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:30:39.033671 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.033658    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "kube-api-access-qfbcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:30:39.033899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.033868    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:30:39.033990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.033897    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:30:39.034339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.034293    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:30:39.038346 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.038240    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted" pod "d58c2f33-be41-4e16-91bc-293ee27d908e" (UID: "d58c2f33-be41-4e16-91bc-293ee27d908e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:30:39.110066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.110026    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:39.132868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132840    5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-image-registry-private-configuration\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.132868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132864    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbcm\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-kube-api-access-qfbcm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.132868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132875    5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d58c2f33-be41-4e16-91bc-293ee27d908e-installation-pull-secrets\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.133061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132884    5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-registry-tls\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.133061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132894    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d58c2f33-be41-4e16-91bc-293ee27d908e-trusted-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.133061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132902    5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d58c2f33-be41-4e16-91bc-293ee27d908e-ca-trust-extracted\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.133061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.132910    5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d58c2f33-be41-4e16-91bc-293ee27d908e-bound-sa-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:39.264906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.264805    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:39.606031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.605946    5060 generic.go:356] "Generic (PLEG): container finished" podID="d58c2f33-be41-4e16-91bc-293ee27d908e" containerID="a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5" exitCode=0
Oct 30 11:30:39.606031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.606004    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7"
Oct 30 11:30:39.606031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.606019    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7" event={"ID":"d58c2f33-be41-4e16-91bc-293ee27d908e","Type":"ContainerDied","Data":"a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5"}
Oct 30 11:30:39.606265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.606080    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f7dc85469-gjnw7" event={"ID":"d58c2f33-be41-4e16-91bc-293ee27d908e","Type":"ContainerDied","Data":"47e08c4f88d9fb0d1146becd6cf6365c28f891594144540db67afb52cf2a82d7"}
Oct 30 11:30:39.606265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.606099    5060 scope.go:122] "RemoveContainer" containerID="a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5"
Oct 30 11:30:39.614669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.614649    5060 scope.go:122] "RemoveContainer" containerID="a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5"
Oct 30 11:30:39.614924 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:39.614903    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5\": container with ID starting with a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5 not found: ID does not exist" containerID="a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5"
Oct 30 11:30:39.614971 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.614933    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5"} err="failed to get container status \"a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5\": rpc error: code = NotFound desc = could not find container \"a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5\": container with ID starting with a2f35ea893a1cc3831641f13166d871fc3475df2e4d8f3c84a6cb7a9f9324cc5 not found: ID does not exist"
Oct 30 11:30:39.670039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.670006    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f7dc85469-gjnw7"]
Oct 30 11:30:39.682383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:39.682355    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f7dc85469-gjnw7"]
Oct 30 11:30:40.331803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:40.331768    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58c2f33-be41-4e16-91bc-293ee27d908e" path="/var/lib/kubelet/pods/d58c2f33-be41-4e16-91bc-293ee27d908e/volumes"
Oct 30 11:30:43.255392 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:43.255354    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:43.411878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:43.411842    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:46.048390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:46.048351    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:46.232393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:46.232361    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:46.701614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:46.701585    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t6jts-b62vt"]
Oct 30 11:30:46.701792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:46.701761    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-t6jts-b62vt" podUID="3c980147-d63b-436d-bff6-5a458249fe5a" containerName="insights-gathering" containerID="cri-o://6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3" gracePeriod=30
Oct 30 11:30:46.863355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:46.863317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:46.946865 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:46.946843    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:30:47.024749 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.024673    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:47.099273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") pod \"3c980147-d63b-436d-bff6-5a458249fe5a\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") "
Oct 30 11:30:47.099727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099311    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") pod \"3c980147-d63b-436d-bff6-5a458249fe5a\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") "
Oct 30 11:30:47.099727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") pod \"3c980147-d63b-436d-bff6-5a458249fe5a\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") "
Oct 30 11:30:47.099727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099391    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") pod \"3c980147-d63b-436d-bff6-5a458249fe5a\" (UID: \"3c980147-d63b-436d-bff6-5a458249fe5a\") "
Oct 30 11:30:47.099727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099582    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path" pod "3c980147-d63b-436d-bff6-5a458249fe5a" (UID: "3c980147-d63b-436d-bff6-5a458249fe5a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:30:47.099727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099673    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle" pod "3c980147-d63b-436d-bff6-5a458249fe5a" (UID: "3c980147-d63b-436d-bff6-5a458249fe5a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:30:47.099922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.099775    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle" pod "3c980147-d63b-436d-bff6-5a458249fe5a" (UID: "3c980147-d63b-436d-bff6-5a458249fe5a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:30:47.101786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.101761    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779" pod "3c980147-d63b-436d-bff6-5a458249fe5a" (UID: "3c980147-d63b-436d-bff6-5a458249fe5a"). InnerVolumeSpecName "kube-api-access-lb779". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:30:47.200397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.200365    5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-service-ca-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:47.200397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.200392    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb779\" (UniqueName: \"kubernetes.io/projected/3c980147-d63b-436d-bff6-5a458249fe5a-kube-api-access-lb779\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:47.200397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.200402    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c980147-d63b-436d-bff6-5a458249fe5a-trusted-ca-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:47.200608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.200413    5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c980147-d63b-436d-bff6-5a458249fe5a-archives-path\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:30:47.476240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.476204    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:47.619581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.619544    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:47.639892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.639861    5060 generic.go:356] "Generic (PLEG): container finished" podID="3c980147-d63b-436d-bff6-5a458249fe5a" containerID="6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3" exitCode=2
Oct 30 11:30:47.640041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.639908    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6jts-b62vt"
Oct 30 11:30:47.640041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.639909    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6jts-b62vt" event={"ID":"3c980147-d63b-436d-bff6-5a458249fe5a","Type":"ContainerDied","Data":"6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3"}
Oct 30 11:30:47.640041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.639971    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6jts-b62vt" event={"ID":"3c980147-d63b-436d-bff6-5a458249fe5a","Type":"ContainerDied","Data":"38bc9cd8c4f142f3930a856615f71a08922cf30ea95380a515121f6bae39a3ee"}
Oct 30 11:30:47.640041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.640000    5060 scope.go:122] "RemoveContainer" containerID="6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3"
Oct 30 11:30:47.648343 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.648327    5060 scope.go:122] "RemoveContainer" containerID="6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3"
Oct 30 11:30:47.648631 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:47.648614    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3\": container with ID starting with 6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3 not found: ID does not exist" containerID="6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3"
Oct 30 11:30:47.648689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.648640    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3"} err="failed to get container status \"6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3\": rpc error: code = NotFound desc = could not find container \"6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3\": container with ID starting with 6599d9971354cf930b19925b75d5c13b39747c356a83774919dae25c5cd8c3a3 not found: ID does not exist"
Oct 30 11:30:47.694301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.694274    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t6jts-b62vt"]
Oct 30 11:30:47.706976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:47.706949    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-t6jts-b62vt"]
Oct 30 11:30:48.332285 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:48.332240    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c980147-d63b-436d-bff6-5a458249fe5a" path="/var/lib/kubelet/pods/3c980147-d63b-436d-bff6-5a458249fe5a/volumes"
Oct 30 11:30:49.855775 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:30:49.855742    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:30:58.256981 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:58.256946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:30:58.414179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:30:58.414144    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:01.701560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:01.701527    5060 generic.go:356] "Generic (PLEG): container finished" podID="c5013b2f-c423-4c36-aaa0-917c35cf23f8" containerID="0fa59d675acc9019d4772fc45f0b3d2b0e146dab7f0b1b3cb78e8e0e2bb2f86d" exitCode=0
Oct 30 11:31:01.702086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:01.701609    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq" event={"ID":"c5013b2f-c423-4c36-aaa0-917c35cf23f8","Type":"ContainerDied","Data":"0fa59d675acc9019d4772fc45f0b3d2b0e146dab7f0b1b3cb78e8e0e2bb2f86d"}
Oct 30 11:31:01.702086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:01.702015    5060 scope.go:122] "RemoveContainer" containerID="0fa59d675acc9019d4772fc45f0b3d2b0e146dab7f0b1b3cb78e8e0e2bb2f86d"
Oct 30 11:31:02.124308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.124193    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc" podUID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" containerName="registry" containerID="cri-o://f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c" gracePeriod=30
Oct 30 11:31:02.367986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.367958    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:31:02.533062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533022    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533142    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533162    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533191    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533216    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533269    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533308    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") pod \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\" (UID: \"bebf069a-0c3e-44c6-9ad8-23dc0056c442\") "
Oct 30 11:31:02.533747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533715    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:31:02.533822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.533756    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:31:02.535792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.535756    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "kube-api-access-cwtms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:31:02.535928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.535792    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:31:02.536099 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.536077    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:31:02.536238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.536219    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:31:02.536323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.536304    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:31:02.540751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.540730    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted" pod "bebf069a-0c3e-44c6-9ad8-23dc0056c442" (UID: "bebf069a-0c3e-44c6-9ad8-23dc0056c442"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:31:02.634790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634757    5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-tls\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.634790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634788    5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-installation-pull-secrets\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.634790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634798    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-trusted-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.635022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634807    5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bebf069a-0c3e-44c6-9ad8-23dc0056c442-ca-trust-extracted\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.635022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634816    5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bebf069a-0c3e-44c6-9ad8-23dc0056c442-registry-certificates\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.635022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634825    5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-bound-sa-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.635022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634833    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtms\" (UniqueName: \"kubernetes.io/projected/bebf069a-0c3e-44c6-9ad8-23dc0056c442-kube-api-access-cwtms\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.635022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.634842    5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bebf069a-0c3e-44c6-9ad8-23dc0056c442-image-registry-private-configuration\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:02.710890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.710852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wvjrq" event={"ID":"c5013b2f-c423-4c36-aaa0-917c35cf23f8","Type":"ContainerStarted","Data":"3aff83330b4c6b571fad3e4c772f6d85c8c951760dd7f63f367427b830c8bd88"}
Oct 30 11:31:02.712113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.712087    5060 generic.go:356] "Generic (PLEG): container finished" podID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" containerID="f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c" exitCode=0
Oct 30 11:31:02.712235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.712136    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc"
Oct 30 11:31:02.712235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.712168    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc" event={"ID":"bebf069a-0c3e-44c6-9ad8-23dc0056c442","Type":"ContainerDied","Data":"f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c"}
Oct 30 11:31:02.712235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.712211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d654c48b4-zcdvc" event={"ID":"bebf069a-0c3e-44c6-9ad8-23dc0056c442","Type":"ContainerDied","Data":"647257c75a2e0ced4b122d62f35ebe5ed4016091cd76525e390f59b942dc5b4b"}
Oct 30 11:31:02.712235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.712230    5060 scope.go:122] "RemoveContainer" containerID="f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c"
Oct 30 11:31:02.720884 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.720868    5060 scope.go:122] "RemoveContainer" containerID="f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c"
Oct 30 11:31:02.721160 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:31:02.721140    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c\": container with ID starting with f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c not found: ID does not exist" containerID="f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c"
Oct 30 11:31:02.721212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.721169    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c"} err="failed to get container status \"f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c\": rpc error: code = NotFound desc = could not find container \"f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c\": container with ID starting with f866d537c8b8caa5c146eed8da8c022162fce5e81b5b9a05532d037bd335286c not found: ID does not exist"
Oct 30 11:31:02.789278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.789202    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d654c48b4-zcdvc"]
Oct 30 11:31:02.801851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:02.801820    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d654c48b4-zcdvc"]
Oct 30 11:31:04.331584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:04.331548    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" path="/var/lib/kubelet/pods/bebf069a-0c3e-44c6-9ad8-23dc0056c442/volumes"
Oct 30 11:31:19.854242 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:31:19.854206    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:31:39.859095 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:39.859011    5060 generic.go:356] "Generic (PLEG): container finished" podID="2822b498-a2ec-4914-9846-1ee191542e12" containerID="3234fcce45cba4825ef444d1e7aec2d94a935ad5a0192431c94938a752c1d15b" exitCode=0
Oct 30 11:31:39.859511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:39.859093    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h" event={"ID":"2822b498-a2ec-4914-9846-1ee191542e12","Type":"ContainerDied","Data":"3234fcce45cba4825ef444d1e7aec2d94a935ad5a0192431c94938a752c1d15b"}
Oct 30 11:31:40.995785 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:40.995763    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:31:41.119674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.119592    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8\" (UniqueName: \"kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8\") pod \"2822b498-a2ec-4914-9846-1ee191542e12\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") "
Oct 30 11:31:41.119674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.119659    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle\") pod \"2822b498-a2ec-4914-9846-1ee191542e12\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") "
Oct 30 11:31:41.119864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.119708    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle\") pod \"2822b498-a2ec-4914-9846-1ee191542e12\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") "
Oct 30 11:31:41.119864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.119748    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path\") pod \"2822b498-a2ec-4914-9846-1ee191542e12\" (UID: \"2822b498-a2ec-4914-9846-1ee191542e12\") "
Oct 30 11:31:41.120118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.120082    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle" pod "2822b498-a2ec-4914-9846-1ee191542e12" (UID: "2822b498-a2ec-4914-9846-1ee191542e12"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:31:41.120118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.120095    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path" pod "2822b498-a2ec-4914-9846-1ee191542e12" (UID: "2822b498-a2ec-4914-9846-1ee191542e12"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:31:41.120219 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.120083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle" pod "2822b498-a2ec-4914-9846-1ee191542e12" (UID: "2822b498-a2ec-4914-9846-1ee191542e12"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:31:41.122036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.122012    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8" pod "2822b498-a2ec-4914-9846-1ee191542e12" (UID: "2822b498-a2ec-4914-9846-1ee191542e12"). InnerVolumeSpecName "kube-api-access-fq6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:31:41.220231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.220196    5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2822b498-a2ec-4914-9846-1ee191542e12-archives-path\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:41.220231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.220225    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6s8\" (UniqueName: \"kubernetes.io/projected/2822b498-a2ec-4914-9846-1ee191542e12-kube-api-access-fq6s8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:41.220428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.220236    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-trusted-ca-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:41.220428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.220251    5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2822b498-a2ec-4914-9846-1ee191542e12-service-ca-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:31:41.867335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.867305    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h"
Oct 30 11:31:41.867562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.867306    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hb6nn-fxr7h" event={"ID":"2822b498-a2ec-4914-9846-1ee191542e12","Type":"ContainerDied","Data":"baa78990d3eee18b84c25d885347b0ca9d7d45408a1c8c8310eacf021067e7d3"}
Oct 30 11:31:41.867562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:41.867434    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa78990d3eee18b84c25d885347b0ca9d7d45408a1c8c8310eacf021067e7d3"
Oct 30 11:31:42.370787 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:42.370754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:42.517903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:42.517870    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:43.561547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:43.561511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:43.774897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:43.774861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:46.185130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:46.185088    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:49.855668 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:31:49.855630    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:31:50.598586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:50.598551    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:51.422491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:51.422442    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:51.610436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:51.610393    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:51.794081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:51.794051    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:51.943796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:51.943754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:31:52.090324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:31:52.090239    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:01.422071 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:01.422035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:01.564085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:01.564054    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:19.856514 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:32:19.856443    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:32:34.822312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:34.822264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:36.658367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:36.658328    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:46.318071 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:46.318035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:47.172695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:47.172656    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:49.856932 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:32:49.856892    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:32:59.528770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:59.528730    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:32:59.674584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:32:59.674547    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:04.904496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:04.904440    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:05.045740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:05.045704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:05.197161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:05.197128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:05.350602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:05.350573    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:12.173335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:12.173293    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:12.319939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:12.319890    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:19.855824 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:33:19.855781    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:33:28.359024 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:28.358987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:28.530060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:28.530022    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:46.457923 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:33:46.457825    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:33:49.855768 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:33:49.855732    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:34:01.788724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:01.788676    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:34:01.929070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:01.929031    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:34:19.856939 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:34:19.856892    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:34:35.304769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:35.304730    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:34:35.461473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:35.461419    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:34:46.596802 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:46.596763    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:34:49.857881 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:34:49.857847    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:34:56.343997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:56.343968    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:34:56.345621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:34:56.345599    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:35:01.344335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:01.344295    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:01.563792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:01.563754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:01.722729 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:01.722694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:03.309214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.309180    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:03.461463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461420    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"]
Oct 30 11:35:03.461758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461745    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" containerName="registry"
Oct 30 11:35:03.461806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461759    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" containerName="registry"
Oct 30 11:35:03.461806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461771    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3c980147-d63b-436d-bff6-5a458249fe5a" containerName="insights-gathering"
Oct 30 11:35:03.461806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461777    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c980147-d63b-436d-bff6-5a458249fe5a" containerName="insights-gathering"
Oct 30 11:35:03.461806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461793    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2822b498-a2ec-4914-9846-1ee191542e12" containerName="insights-gathering"
Oct 30 11:35:03.461806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461798    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2822b498-a2ec-4914-9846-1ee191542e12" containerName="insights-gathering"
Oct 30 11:35:03.461950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461814    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d58c2f33-be41-4e16-91bc-293ee27d908e" containerName="registry"
Oct 30 11:35:03.461950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461819    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58c2f33-be41-4e16-91bc-293ee27d908e" containerName="registry"
Oct 30 11:35:03.461950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461871    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2822b498-a2ec-4914-9846-1ee191542e12" containerName="insights-gathering"
Oct 30 11:35:03.461950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461881    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58c2f33-be41-4e16-91bc-293ee27d908e" containerName="registry"
Oct 30 11:35:03.461950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461887    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebf069a-0c3e-44c6-9ad8-23dc0056c442" containerName="registry"
Oct 30 11:35:03.461950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.461893    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c980147-d63b-436d-bff6-5a458249fe5a" containerName="insights-gathering"
Oct 30 11:35:03.464990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.464973    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.485367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.485336    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 11:35:03.485595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.485497    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 11:35:03.485771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.485663    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hzp9d\""
Oct 30 11:35:03.487072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.487048    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"]
Oct 30 11:35:03.487826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.487795    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:03.513263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.513230    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.513431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.513281    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwkh\" (UniqueName: \"kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.513495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.513417    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.597431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.597354    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"]
Oct 30 11:35:03.600767 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.600748    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.615527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.615435    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwkh\" (UniqueName: \"kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.615701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.615574    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.615701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.615625    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.616033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.616012    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.616794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.616769    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.621143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.621119    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-2zmfn\""
Oct 30 11:35:03.622231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.622208    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"]
Oct 30 11:35:03.657723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.657687    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwkh\" (UniqueName: \"kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.716496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.716432    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2cvp\" (UniqueName: \"kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.716686 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.716523    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.716686 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.716552    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.781148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.781119    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:03.817761 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.817714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.817957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.817795    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.817957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.817853    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2cvp\" (UniqueName: \"kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.818111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.818092    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.818154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.818121    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.847094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.846949    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2cvp\" (UniqueName: \"kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:03.917578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:03.917487    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:04.080181 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:04.080152    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"]
Oct 30 11:35:04.081270 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:04.081246    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34bd99c_7bd9_4aea_9969_840ff39ee70c.slice/crio-758318319cf85b6e318c1d6312d2b5c8668360c0962199eeed39f1f703726722 WatchSource:0}: Error finding container 758318319cf85b6e318c1d6312d2b5c8668360c0962199eeed39f1f703726722: Status 404 returned error can't find the container with id 758318319cf85b6e318c1d6312d2b5c8668360c0962199eeed39f1f703726722
Oct 30 11:35:04.083171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:04.083152    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:35:04.217102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:04.217076    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"]
Oct 30 11:35:04.218265 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:04.218240    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b11fdfd_8734_44f2_ab59_eb247f02c155.slice/crio-d8ae5b0e46ae39c1ce80e1b6e1c4a31e6e5a430ce7ff4ac434bf4881b67699b1 WatchSource:0}: Error finding container d8ae5b0e46ae39c1ce80e1b6e1c4a31e6e5a430ce7ff4ac434bf4881b67699b1: Status 404 returned error can't find the container with id d8ae5b0e46ae39c1ce80e1b6e1c4a31e6e5a430ce7ff4ac434bf4881b67699b1
Oct 30 11:35:04.631253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:04.631112    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx" event={"ID":"3b11fdfd-8734-44f2-ab59-eb247f02c155","Type":"ContainerStarted","Data":"d8ae5b0e46ae39c1ce80e1b6e1c4a31e6e5a430ce7ff4ac434bf4881b67699b1"}
Oct 30 11:35:04.632933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:04.632900    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" event={"ID":"d34bd99c-7bd9-4aea-9969-840ff39ee70c","Type":"ContainerStarted","Data":"758318319cf85b6e318c1d6312d2b5c8668360c0962199eeed39f1f703726722"}
Oct 30 11:35:04.744116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:04.744077    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:05.873995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.872144    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"]
Oct 30 11:35:05.876316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.876289    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:05.904053 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.904019    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gp4rk\""
Oct 30 11:35:05.905869 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.905840    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"]
Oct 30 11:35:05.936717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.936681    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jhm\" (UniqueName: \"kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:05.936912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.936741    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:05.936912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:05.936859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.038126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.038090    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jhm\" (UniqueName: \"kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.038277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.038135    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.038277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.038166    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.038542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.038522    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.038769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.038744    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.064693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.064631    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jhm\" (UniqueName: \"kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.197844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.197805    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:06.515341 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.515317    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"]
Oct 30 11:35:06.518067 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:06.518027    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b94ea0c_e42e_45b6_b99d_a019b4bade19.slice/crio-575f3169cc5848690efa2c68e72a8efc498718983956269f20c34523ccc27000 WatchSource:0}: Error finding container 575f3169cc5848690efa2c68e72a8efc498718983956269f20c34523ccc27000: Status 404 returned error can't find the container with id 575f3169cc5848690efa2c68e72a8efc498718983956269f20c34523ccc27000
Oct 30 11:35:06.643697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:06.643656    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84" event={"ID":"1b94ea0c-e42e-45b6-b99d-a019b4bade19","Type":"ContainerStarted","Data":"575f3169cc5848690efa2c68e72a8efc498718983956269f20c34523ccc27000"}
Oct 30 11:35:07.150051 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:07.150012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:07.306438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:07.306396    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:08.008239 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.008204    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"]
Oct 30 11:35:08.012542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.012509    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.041029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.040994    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fr7sp\""
Oct 30 11:35:08.042233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.042209    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"]
Oct 30 11:35:08.056052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.056013    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.056220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.056110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6nhp\" (UniqueName: \"kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.056220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.056145    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.104418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.104377    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"]
Oct 30 11:35:08.109218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.108620    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.127695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.127618    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"]
Oct 30 11:35:08.129345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.129317    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-j6qbt\""
Oct 30 11:35:08.157732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.157697    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.158225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.157844    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.158225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.157949    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6nhp\" (UniqueName: \"kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.158225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.158026    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.158225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.158053    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhs6l\" (UniqueName: \"kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.158225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.158086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.158225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.158122    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.158591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.158392    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.184986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.184923    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6nhp\" (UniqueName: \"kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.214869 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.214837    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"]
Oct 30 11:35:08.220292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.219665    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.246334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.246300    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"]
Oct 30 11:35:08.256357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.256329    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-cd2gf\""
Oct 30 11:35:08.258620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.258560    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.258736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.258639    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.258736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.258696    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdmr\" (UniqueName: \"kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.258907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.258787    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.258907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.258826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.258907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.258863    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhs6l\" (UniqueName: \"kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.259066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.259005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.259271 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.259245    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.309754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.309722    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhs6l\" (UniqueName: \"kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.334192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.334165    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:08.360553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.360427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdmr\" (UniqueName: \"kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.361314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.360921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.361314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.361012    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.361314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.361188    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.361314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.361284    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.386069 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.386003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdmr\" (UniqueName: \"kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.432125 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.432090    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:08.540841 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.540696    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:08.550929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.550909    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"]
Oct 30 11:35:08.554584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.554554    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.575704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.574946    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5qmrg\""
Oct 30 11:35:08.576096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.576077    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 11:35:08.578202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.578177    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"]
Oct 30 11:35:08.653726 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.653695    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"]
Oct 30 11:35:08.656210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.655896    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:08.664045 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664020    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664054    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664180    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664267 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664254    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664288    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664319    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664472    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664511    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnzj\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.664570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.664547    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.681983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.681940    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"]
Oct 30 11:35:08.694113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.693069    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.709801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.709743    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-4ccr4\""
Oct 30 11:35:08.711518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.710975    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"]
Oct 30 11:35:08.745171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.745132    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"]
Oct 30 11:35:08.748978 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:08.748944    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6093080c_661e_48b3_b549_cd187ebc9ffc.slice/crio-47bafab79d720e8c675604b73f4a71c7317bbfddc7abf1240aa28a6650be9faf WatchSource:0}: Error finding container 47bafab79d720e8c675604b73f4a71c7317bbfddc7abf1240aa28a6650be9faf: Status 404 returned error can't find the container with id 47bafab79d720e8c675604b73f4a71c7317bbfddc7abf1240aa28a6650be9faf
Oct 30 11:35:08.765575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765528    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765588    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765619    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765637    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765655    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765686    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765720    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.765763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765758    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765789    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765826    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765884    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9ck\" (UniqueName: \"kubernetes.io/projected/33aa61d0-08cb-41ba-92ef-0c2ecd809355-kube-api-access-cn9ck\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.765968    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnzj\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.766009    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.766037    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.766075    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.766099    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.766707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.766643    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.767106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.767081    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.767922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.767837    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.768216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.768195    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.768447 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.768422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.769680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.769659    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.771223 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.771199    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.791672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.791530    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnzj\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.792868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.792835    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmtjm\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.811861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.811811    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:08.866935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.866875    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.866935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.866942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.866976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867000    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867030    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867051    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867075    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867103    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9ck\" (UniqueName: \"kubernetes.io/projected/33aa61d0-08cb-41ba-92ef-0c2ecd809355-kube-api-access-cn9ck\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867503    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.867760 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.867738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.868820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.868793    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.870296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.870227    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"]
Oct 30 11:35:08.870789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.870768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.870911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.870837    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.870999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.870978    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.871955 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:08.871928    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bff443_e933_420d_b91a_7d537c54b376.slice/crio-145cddd4569c0c0aeacdc87719eb98265ee68bc9a33a66a992c95b86e76c76f8 WatchSource:0}: Error finding container 145cddd4569c0c0aeacdc87719eb98265ee68bc9a33a66a992c95b86e76c76f8: Status 404 returned error can't find the container with id 145cddd4569c0c0aeacdc87719eb98265ee68bc9a33a66a992c95b86e76c76f8
Oct 30 11:35:08.872078 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.872063    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:08.872964 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.872933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.895962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.895925    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9ck\" (UniqueName: \"kubernetes.io/projected/33aa61d0-08cb-41ba-92ef-0c2ecd809355-kube-api-access-cn9ck\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:08.897420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:08.897389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa61d0-08cb-41ba-92ef-0c2ecd809355-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn\" (UID: \"33aa61d0-08cb-41ba-92ef-0c2ecd809355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:09.018943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.018904    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:09.205668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.203796    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"]
Oct 30 11:35:09.212995 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:09.212955    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069c84b3_658d_4e00_a7f1_aebbe5173974.slice/crio-f9fc6fdf4811e17d2f19d8aca5aec14cc31a09a9969b760e63df6278490517bf WatchSource:0}: Error finding container f9fc6fdf4811e17d2f19d8aca5aec14cc31a09a9969b760e63df6278490517bf: Status 404 returned error can't find the container with id f9fc6fdf4811e17d2f19d8aca5aec14cc31a09a9969b760e63df6278490517bf
Oct 30 11:35:09.350917 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.350268    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"]
Oct 30 11:35:09.355877 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:09.355835    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aa61d0_08cb_41ba_92ef_0c2ecd809355.slice/crio-71cf0556af00516647a9c14a6705f5726b2dd1795e48c3f1f703a5910b230e3f WatchSource:0}: Error finding container 71cf0556af00516647a9c14a6705f5726b2dd1795e48c3f1f703a5910b230e3f: Status 404 returned error can't find the container with id 71cf0556af00516647a9c14a6705f5726b2dd1795e48c3f1f703a5910b230e3f
Oct 30 11:35:09.668177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.668036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl" event={"ID":"6093080c-661e-48b3-b549-cd187ebc9ffc","Type":"ContainerStarted","Data":"47bafab79d720e8c675604b73f4a71c7317bbfddc7abf1240aa28a6650be9faf"}
Oct 30 11:35:09.672601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.672512    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn" event={"ID":"33aa61d0-08cb-41ba-92ef-0c2ecd809355","Type":"ContainerStarted","Data":"71cf0556af00516647a9c14a6705f5726b2dd1795e48c3f1f703a5910b230e3f"}
Oct 30 11:35:09.675136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.675066    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" event={"ID":"069c84b3-658d-4e00-a7f1-aebbe5173974","Type":"ContainerStarted","Data":"f9fc6fdf4811e17d2f19d8aca5aec14cc31a09a9969b760e63df6278490517bf"}
Oct 30 11:35:09.677218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.677137    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k" event={"ID":"dac7c339-9cca-4655-ab42-1d1a13615408","Type":"ContainerStarted","Data":"3700caa7b2504d1af9a8050a4140974f90a4f7ff89c536d4f6b169dce86f1054"}
Oct 30 11:35:09.680174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:09.680098    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz" event={"ID":"d8bff443-e933-420d-b91a-7d537c54b376","Type":"ContainerStarted","Data":"145cddd4569c0c0aeacdc87719eb98265ee68bc9a33a66a992c95b86e76c76f8"}
Oct 30 11:35:11.950566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:11.950514    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:14.721942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:14.721910    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerID="36f43dc5a2727dc95aec43a9a2d7cbf3b404f645ce8785c8df3629d9841834ef" exitCode=0
Oct 30 11:35:14.722549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:14.722001    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx" event={"ID":"3b11fdfd-8734-44f2-ab59-eb247f02c155","Type":"ContainerDied","Data":"36f43dc5a2727dc95aec43a9a2d7cbf3b404f645ce8785c8df3629d9841834ef"}
Oct 30 11:35:14.725436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:14.725408    5060 generic.go:356] "Generic (PLEG): container finished" podID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerID="7e185489b803bf1c30a9425e95ba10a98a7f6ba4b8e122d927afaadbf97d9039" exitCode=0
Oct 30 11:35:14.725551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:14.725535    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" event={"ID":"d34bd99c-7bd9-4aea-9969-840ff39ee70c","Type":"ContainerDied","Data":"7e185489b803bf1c30a9425e95ba10a98a7f6ba4b8e122d927afaadbf97d9039"}
Oct 30 11:35:15.070999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.070966    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"]
Oct 30 11:35:15.075331 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.075226    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.094618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.094543    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"]
Oct 30 11:35:15.233937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.233861    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxl8\" (UniqueName: \"kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.234123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.233951    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.234123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.233998    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.335364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.335294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxl8\" (UniqueName: \"kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.335564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.335367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.335564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.335401    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.335733 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.335717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.336218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.336199    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.372438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.372373    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxl8\" (UniqueName: \"kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.404762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.404729    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:15.736924 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.736886    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerID="8778e7e680acdf96a4c379d9cf6c2a20206d462275816003b8835ef3749fd1e5" exitCode=0
Oct 30 11:35:15.736924 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.736963    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx" event={"ID":"3b11fdfd-8734-44f2-ab59-eb247f02c155","Type":"ContainerDied","Data":"8778e7e680acdf96a4c379d9cf6c2a20206d462275816003b8835ef3749fd1e5"}
Oct 30 11:35:15.767677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.767557    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"]
Oct 30 11:35:15.998552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:15.997326    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"]
Oct 30 11:35:16.004111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.003438    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.019789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.019731    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"]
Oct 30 11:35:16.033473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.033436    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"]
Oct 30 11:35:16.046696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.046676    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.058342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.058315    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"]
Oct 30 11:35:16.070387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.070360    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"]
Oct 30 11:35:16.076304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.076281    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.093794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.093769    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"]
Oct 30 11:35:16.094037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.094002    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jgxs8\""
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145194    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145242    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkc8d\" (UniqueName: \"kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145294    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145351    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145383    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145412    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145438    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhc5p\" (UniqueName: \"kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145484    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.145711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.145530    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442vr\" (UniqueName: \"kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.174552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.174514    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:35:16.174685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.174615    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:35:16.174685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.174657    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:35:16.246416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246381    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.246598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246432    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkc8d\" (UniqueName: \"kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.246598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246500    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.246598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246559    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.246598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246589    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.246808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246617    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.246808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhc5p\" (UniqueName: \"kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.246808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.246808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.246713    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442vr\" (UniqueName: \"kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.247381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.247356    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.247921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.247893    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.248221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.248196    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.248549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.248528    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.248822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.248763    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.249176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.249158    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.272355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.272328    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhc5p\" (UniqueName: \"kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.273960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.273380    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkc8d\" (UniqueName: \"kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.273960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.273797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-442vr\" (UniqueName: \"kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.353369 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.353333    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:16.371357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.371324    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:16.400139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.399958    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:16.713718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.713683    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"]
Oct 30 11:35:16.715642 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:16.715608    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730f5ac7_70ed_4af4_b93b_126af23f43ff.slice/crio-8488976a2b1255cc8757a94b2693f0851a18f64e723fc9311c943abe38ab119a WatchSource:0}: Error finding container 8488976a2b1255cc8757a94b2693f0851a18f64e723fc9311c943abe38ab119a: Status 404 returned error can't find the container with id 8488976a2b1255cc8757a94b2693f0851a18f64e723fc9311c943abe38ab119a
Oct 30 11:35:16.759717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.758508    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerID="fb798dd9b93230e5e163edb32666f2715b053b27da455acc0b1ce5a31c1b861e" exitCode=0
Oct 30 11:35:16.759717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.758619    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl" event={"ID":"1f647cd7-39c3-4094-8439-3d221fbc4eec","Type":"ContainerDied","Data":"fb798dd9b93230e5e163edb32666f2715b053b27da455acc0b1ce5a31c1b861e"}
Oct 30 11:35:16.759717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.758665    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl" event={"ID":"1f647cd7-39c3-4094-8439-3d221fbc4eec","Type":"ContainerStarted","Data":"6a4e0030dce878f4bfad6a855da3e39d815c3ce787df2f17d28cd6f1b02e56d7"}
Oct 30 11:35:16.767313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.766650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn" event={"ID":"33aa61d0-08cb-41ba-92ef-0c2ecd809355","Type":"ContainerStarted","Data":"e659184ee12e3e8a44a8c7cb504e273b065aeda4c3f5ceed908195f4cbc50ccb"}
Oct 30 11:35:16.770572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.770337    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb" event={"ID":"730f5ac7-70ed-4af4-b93b-126af23f43ff","Type":"ContainerStarted","Data":"8488976a2b1255cc8757a94b2693f0851a18f64e723fc9311c943abe38ab119a"}
Oct 30 11:35:16.770641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.770603    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"]
Oct 30 11:35:16.779961 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:16.777492    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287868b1_74f6_4f01_9949_c58ce8078a57.slice/crio-acd8bb84a64246f3467708110f9066c74df49a4c1a067207dc923450fa9a3609 WatchSource:0}: Error finding container acd8bb84a64246f3467708110f9066c74df49a4c1a067207dc923450fa9a3609: Status 404 returned error can't find the container with id acd8bb84a64246f3467708110f9066c74df49a4c1a067207dc923450fa9a3609
Oct 30 11:35:16.779961 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.779847    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"]
Oct 30 11:35:16.781180 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:16.781146    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb1da02_1374_418b_9288_314da3ff59c7.slice/crio-75d62a8caef59c38442793e38f182d0ca9853dbd7d2d04cc50afe37b30015d8e WatchSource:0}: Error finding container 75d62a8caef59c38442793e38f182d0ca9853dbd7d2d04cc50afe37b30015d8e: Status 404 returned error can't find the container with id 75d62a8caef59c38442793e38f182d0ca9853dbd7d2d04cc50afe37b30015d8e
Oct 30 11:35:16.811530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.811491    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:35:16.811636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.811577    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:35:16.811636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:16.811626    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:35:17.019617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.019542    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:17.025495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.025465    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:17.093394 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.093340    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn" podStartSLOduration=2.281096722 podStartE2EDuration="9.093320691s" totalImagesPullingTime="6.812223969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:16.852715929 +0000 UTC m=+320.647600423" watchObservedRunningTime="2026-10-30 11:35:17.093320691 +0000 UTC m=+320.888205188"
Oct 30 11:35:17.781481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.781082    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" event={"ID":"069c84b3-658d-4e00-a7f1-aebbe5173974","Type":"ContainerStarted","Data":"c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"}
Oct 30 11:35:17.790179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.789323    5060 generic.go:356] "Generic (PLEG): container finished" podID="287868b1-74f6-4f01-9949-c58ce8078a57" containerID="bf708d92fc86215355167266f0877d05f0e0a0a5049187198f1b1f7e8e64befc" exitCode=0
Oct 30 11:35:17.790179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.789426    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" event={"ID":"287868b1-74f6-4f01-9949-c58ce8078a57","Type":"ContainerDied","Data":"bf708d92fc86215355167266f0877d05f0e0a0a5049187198f1b1f7e8e64befc"}
Oct 30 11:35:17.790179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.789504    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" event={"ID":"287868b1-74f6-4f01-9949-c58ce8078a57","Type":"ContainerStarted","Data":"acd8bb84a64246f3467708110f9066c74df49a4c1a067207dc923450fa9a3609"}
Oct 30 11:35:17.804895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.801556    5060 generic.go:356] "Generic (PLEG): container finished" podID="fdb1da02-1374-418b-9288-314da3ff59c7" containerID="9126070c2b12d58e60217ab8ecb6ea222418d8f72896fa98ee05dab85f4be9e4" exitCode=0
Oct 30 11:35:17.804895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.801790    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf" event={"ID":"fdb1da02-1374-418b-9288-314da3ff59c7","Type":"ContainerDied","Data":"9126070c2b12d58e60217ab8ecb6ea222418d8f72896fa98ee05dab85f4be9e4"}
Oct 30 11:35:17.804895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.801877    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf" event={"ID":"fdb1da02-1374-418b-9288-314da3ff59c7","Type":"ContainerStarted","Data":"75d62a8caef59c38442793e38f182d0ca9853dbd7d2d04cc50afe37b30015d8e"}
Oct 30 11:35:17.804895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.804518    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerID="4cfd627b9ac88ece19bbc592769bdf6764a724fc79dc73e9892520097e0ea369" exitCode=0
Oct 30 11:35:17.804895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.804695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl" event={"ID":"1f647cd7-39c3-4094-8439-3d221fbc4eec","Type":"ContainerDied","Data":"4cfd627b9ac88ece19bbc592769bdf6764a724fc79dc73e9892520097e0ea369"}
Oct 30 11:35:17.818950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.818919    5060 generic.go:356] "Generic (PLEG): container finished" podID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerID="914fd8de5c77cf816a3329e0e7ddc0b180ecbc4965246889560e23f0e25bf033" exitCode=0
Oct 30 11:35:17.819192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.818993    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb" event={"ID":"730f5ac7-70ed-4af4-b93b-126af23f43ff","Type":"ContainerDied","Data":"914fd8de5c77cf816a3329e0e7ddc0b180ecbc4965246889560e23f0e25bf033"}
Oct 30 11:35:17.819674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.819618    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:17.820843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.820816    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn"
Oct 30 11:35:17.836974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.836904    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podStartSLOduration=2.241829017 podStartE2EDuration="9.836883839s" totalImagesPullingTime="7.595054822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:17.835837763 +0000 UTC m=+321.630722257" watchObservedRunningTime="2026-10-30 11:35:17.836883839 +0000 UTC m=+321.631768334"
Oct 30 11:35:17.872876 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.872846    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:17.879548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:17.879414    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:18.836476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.836224    5060 generic.go:356] "Generic (PLEG): container finished" podID="dac7c339-9cca-4655-ab42-1d1a13615408" containerID="10835481dc988babb8900fd9f57c5b86146bc8e5ffdf418d3c7cd857ff51ef3d" exitCode=0
Oct 30 11:35:18.836476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.836324    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k" event={"ID":"dac7c339-9cca-4655-ab42-1d1a13615408","Type":"ContainerDied","Data":"10835481dc988babb8900fd9f57c5b86146bc8e5ffdf418d3c7cd857ff51ef3d"}
Oct 30 11:35:18.842731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.842004    5060 generic.go:356] "Generic (PLEG): container finished" podID="287868b1-74f6-4f01-9949-c58ce8078a57" containerID="dbd8b5ab649d0a2dfcd2bcb114b57e6950d1eccb37e4fe5a356a28d53fb7f14b" exitCode=0
Oct 30 11:35:18.842731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.842095    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" event={"ID":"287868b1-74f6-4f01-9949-c58ce8078a57","Type":"ContainerDied","Data":"dbd8b5ab649d0a2dfcd2bcb114b57e6950d1eccb37e4fe5a356a28d53fb7f14b"}
Oct 30 11:35:18.847140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.846496    5060 generic.go:356] "Generic (PLEG): container finished" podID="d8bff443-e933-420d-b91a-7d537c54b376" containerID="fdcdff9b366ebbde4c2bcf984b1dd5286dc9b4d0f440d44bbc8eb13c26d302c1" exitCode=0
Oct 30 11:35:18.847140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.846635    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz" event={"ID":"d8bff443-e933-420d-b91a-7d537c54b376","Type":"ContainerDied","Data":"fdcdff9b366ebbde4c2bcf984b1dd5286dc9b4d0f440d44bbc8eb13c26d302c1"}
Oct 30 11:35:18.854734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.850996    5060 generic.go:356] "Generic (PLEG): container finished" podID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerID="eb83bcf3750e20946a5317626ab4c2c09978d7843dd3365eaf7de9c6bb17870b" exitCode=0
Oct 30 11:35:18.854734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.852750    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl" event={"ID":"6093080c-661e-48b3-b549-cd187ebc9ffc","Type":"ContainerDied","Data":"eb83bcf3750e20946a5317626ab4c2c09978d7843dd3365eaf7de9c6bb17870b"}
Oct 30 11:35:18.854734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.852801    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:18.854906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:18.854830    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:35:19.865700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.865663    5060 generic.go:356] "Generic (PLEG): container finished" podID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerID="3f891c6cf01420f958f5fc004c5b0eba1c89635ea0ad2fb8b18715b7eeb8de75" exitCode=0
Oct 30 11:35:19.866138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.865779    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl" event={"ID":"6093080c-661e-48b3-b549-cd187ebc9ffc","Type":"ContainerDied","Data":"3f891c6cf01420f958f5fc004c5b0eba1c89635ea0ad2fb8b18715b7eeb8de75"}
Oct 30 11:35:19.874481 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:35:19.872382    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:35:19.879137 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.879108    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84" event={"ID":"1b94ea0c-e42e-45b6-b99d-a019b4bade19","Type":"ContainerStarted","Data":"1905af64796e56c8d56edf370d5db784e028479d19c80162b75ba3b39969c889"}
Oct 30 11:35:19.883335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.883299    5060 generic.go:356] "Generic (PLEG): container finished" podID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerID="876784cc4e8d2f272a4ffddfcc42d71028410a19979a9edb0ae196df25c812d0" exitCode=0
Oct 30 11:35:19.884191 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.883571    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb" event={"ID":"730f5ac7-70ed-4af4-b93b-126af23f43ff","Type":"ContainerDied","Data":"876784cc4e8d2f272a4ffddfcc42d71028410a19979a9edb0ae196df25c812d0"}
Oct 30 11:35:19.886253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.886226    5060 generic.go:356] "Generic (PLEG): container finished" podID="dac7c339-9cca-4655-ab42-1d1a13615408" containerID="6678319d5d080e979cfe674566212a320ff41a28d1b3ad82a6e300bac09278e6" exitCode=0
Oct 30 11:35:19.886348 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.886329    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k" event={"ID":"dac7c339-9cca-4655-ab42-1d1a13615408","Type":"ContainerDied","Data":"6678319d5d080e979cfe674566212a320ff41a28d1b3ad82a6e300bac09278e6"}
Oct 30 11:35:19.890948 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.890913    5060 generic.go:356] "Generic (PLEG): container finished" podID="d8bff443-e933-420d-b91a-7d537c54b376" containerID="a9f68b2e25c906c69f9287ee76aae37aa770f76c6a0a52831c3588cc7fbe553b" exitCode=0
Oct 30 11:35:19.891060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.890994    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz" event={"ID":"d8bff443-e933-420d-b91a-7d537c54b376","Type":"ContainerDied","Data":"a9f68b2e25c906c69f9287ee76aae37aa770f76c6a0a52831c3588cc7fbe553b"}
Oct 30 11:35:19.897204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.896336    5060 generic.go:356] "Generic (PLEG): container finished" podID="fdb1da02-1374-418b-9288-314da3ff59c7" containerID="1656ddf55b3debfeabdb3e2e1e56a4b349dba49f18fc6a73d389befa5058902b" exitCode=0
Oct 30 11:35:19.897204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.896508    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf" event={"ID":"fdb1da02-1374-418b-9288-314da3ff59c7","Type":"ContainerDied","Data":"1656ddf55b3debfeabdb3e2e1e56a4b349dba49f18fc6a73d389befa5058902b"}
Oct 30 11:35:19.902612 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.902581    5060 generic.go:356] "Generic (PLEG): container finished" podID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerID="cf6b04073bbc3068df945ffc74c6db4b5f93429f2fece2a64f38b93ce041582e" exitCode=0
Oct 30 11:35:19.903610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:19.903548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" event={"ID":"d34bd99c-7bd9-4aea-9969-840ff39ee70c","Type":"ContainerDied","Data":"cf6b04073bbc3068df945ffc74c6db4b5f93429f2fece2a64f38b93ce041582e"}
Oct 30 11:35:20.928138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:20.926478    5060 generic.go:356] "Generic (PLEG): container finished" podID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerID="1905af64796e56c8d56edf370d5db784e028479d19c80162b75ba3b39969c889" exitCode=0
Oct 30 11:35:20.928138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:20.927803    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84" event={"ID":"1b94ea0c-e42e-45b6-b99d-a019b4bade19","Type":"ContainerDied","Data":"1905af64796e56c8d56edf370d5db784e028479d19c80162b75ba3b39969c889"}
Oct 30 11:35:29.056151 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:29.056099    5060 generic.go:356] "Generic (PLEG): container finished" podID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerID="3a6f090d6b394d3cfd9b8c28b002339444bfb82b6ae3917012b28f7655b6e00c" exitCode=0
Oct 30 11:35:29.056890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:29.056274    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84" event={"ID":"1b94ea0c-e42e-45b6-b99d-a019b4bade19","Type":"ContainerDied","Data":"3a6f090d6b394d3cfd9b8c28b002339444bfb82b6ae3917012b28f7655b6e00c"}
Oct 30 11:35:30.594421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:30.594357    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:31.622240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:31.622176    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:34.730416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:34.730320    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:35.019354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:35.019277    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:35.172398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:35.172362    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:35.749509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:35.749396    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:36.089512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:36.089423    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:36.151271 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:36.151217    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerID="233c77b4ec00cbf7d4041a6ac6a7509c83a4ea09c9e84cc399d127855d6efa63" exitCode=0
Oct 30 11:35:36.151484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:36.151331    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx" event={"ID":"3b11fdfd-8734-44f2-ab59-eb247f02c155","Type":"ContainerDied","Data":"233c77b4ec00cbf7d4041a6ac6a7509c83a4ea09c9e84cc399d127855d6efa63"}
Oct 30 11:35:36.357328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:36.357226    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:37.344677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.344637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:37.387835 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.387494    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:37.511846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.510879    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle\") pod \"3b11fdfd-8734-44f2-ab59-eb247f02c155\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") "
Oct 30 11:35:37.511846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.510990    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util\") pod \"3b11fdfd-8734-44f2-ab59-eb247f02c155\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") "
Oct 30 11:35:37.511846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.511043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp\" (UniqueName: \"kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp\") pod \"3b11fdfd-8734-44f2-ab59-eb247f02c155\" (UID: \"3b11fdfd-8734-44f2-ab59-eb247f02c155\") "
Oct 30 11:35:37.513234 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.513166    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle" pod "3b11fdfd-8734-44f2-ab59-eb247f02c155" (UID: "3b11fdfd-8734-44f2-ab59-eb247f02c155"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:37.519360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.519331    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util" pod "3b11fdfd-8734-44f2-ab59-eb247f02c155" (UID: "3b11fdfd-8734-44f2-ab59-eb247f02c155"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:37.523135 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.523093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp" pod "3b11fdfd-8734-44f2-ab59-eb247f02c155" (UID: "3b11fdfd-8734-44f2-ab59-eb247f02c155"). InnerVolumeSpecName "kube-api-access-d2cvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:37.575648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.575553    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:37.612167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.612131    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:37.612167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.612166    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b11fdfd-8734-44f2-ab59-eb247f02c155-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:37.612409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:37.612178    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2cvp\" (UniqueName: \"kubernetes.io/projected/3b11fdfd-8734-44f2-ab59-eb247f02c155-kube-api-access-d2cvp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:38.195829 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.195797    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx" event={"ID":"3b11fdfd-8734-44f2-ab59-eb247f02c155","Type":"ContainerDied","Data":"d8ae5b0e46ae39c1ce80e1b6e1c4a31e6e5a430ce7ff4ac434bf4881b67699b1"}
Oct 30 11:35:38.195996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.195861    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8ae5b0e46ae39c1ce80e1b6e1c4a31e6e5a430ce7ff4ac434bf4881b67699b1"
Oct 30 11:35:38.195996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.195928    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d6tqx"
Oct 30 11:35:38.202715 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.202085    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerID="2e2709815b0323cf6c926294fc13a659f36a2d4fa9a6adfc8cb6feeea621718e" exitCode=0
Oct 30 11:35:38.202715 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.202273    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl" event={"ID":"1f647cd7-39c3-4094-8439-3d221fbc4eec","Type":"ContainerDied","Data":"2e2709815b0323cf6c926294fc13a659f36a2d4fa9a6adfc8cb6feeea621718e"}
Oct 30 11:35:38.208639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.208607    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" event={"ID":"287868b1-74f6-4f01-9949-c58ce8078a57","Type":"ContainerStarted","Data":"45c4567ce81fee06cbcc3882a9db5aa73dd004973cdbae09970eea6e5ff98bd5"}
Oct 30 11:35:38.320723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.320178    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" podStartSLOduration=2.616663415 podStartE2EDuration="22.320160011s" totalImagesPullingTime="19.703496596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:38.318666014 +0000 UTC m=+342.113550509" watchObservedRunningTime="2026-10-30 11:35:38.320160011 +0000 UTC m=+342.115044525"
Oct 30 11:35:38.868557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:38.868491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:39.105511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.105443    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:39.222589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.221818    5060 generic.go:356] "Generic (PLEG): container finished" podID="287868b1-74f6-4f01-9949-c58ce8078a57" containerID="45c4567ce81fee06cbcc3882a9db5aa73dd004973cdbae09970eea6e5ff98bd5" exitCode=0
Oct 30 11:35:39.222589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.222523    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" event={"ID":"287868b1-74f6-4f01-9949-c58ce8078a57","Type":"ContainerDied","Data":"45c4567ce81fee06cbcc3882a9db5aa73dd004973cdbae09970eea6e5ff98bd5"}
Oct 30 11:35:39.459585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.459539    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:39.529978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.529886    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util\") pod \"1f647cd7-39c3-4094-8439-3d221fbc4eec\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") "
Oct 30 11:35:39.530157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.529986    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8\" (UniqueName: \"kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8\") pod \"1f647cd7-39c3-4094-8439-3d221fbc4eec\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") "
Oct 30 11:35:39.530157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.530041    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle\") pod \"1f647cd7-39c3-4094-8439-3d221fbc4eec\" (UID: \"1f647cd7-39c3-4094-8439-3d221fbc4eec\") "
Oct 30 11:35:39.532840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.532788    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle" pod "1f647cd7-39c3-4094-8439-3d221fbc4eec" (UID: "1f647cd7-39c3-4094-8439-3d221fbc4eec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:39.537373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.537328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util" pod "1f647cd7-39c3-4094-8439-3d221fbc4eec" (UID: "1f647cd7-39c3-4094-8439-3d221fbc4eec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:39.544439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.544405    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8" pod "1f647cd7-39c3-4094-8439-3d221fbc4eec" (UID: "1f647cd7-39c3-4094-8439-3d221fbc4eec"). InnerVolumeSpecName "kube-api-access-mpxl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:39.631314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.631280    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpxl8\" (UniqueName: \"kubernetes.io/projected/1f647cd7-39c3-4094-8439-3d221fbc4eec-kube-api-access-mpxl8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:39.631314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.631314    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:39.631565 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.631329    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f647cd7-39c3-4094-8439-3d221fbc4eec-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:39.776966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.776905    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:39.993588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:39.993551    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:40.232956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.232917    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" event={"ID":"d34bd99c-7bd9-4aea-9969-840ff39ee70c","Type":"ContainerStarted","Data":"6e79525241735fd92e97189b3a239923e6b748fdbd0b075f5936a52a083833bb"}
Oct 30 11:35:40.236985 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.236954    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl" event={"ID":"1f647cd7-39c3-4094-8439-3d221fbc4eec","Type":"ContainerDied","Data":"6a4e0030dce878f4bfad6a855da3e39d815c3ce787df2f17d28cd6f1b02e56d7"}
Oct 30 11:35:40.237132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.236998    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4e0030dce878f4bfad6a855da3e39d815c3ce787df2f17d28cd6f1b02e56d7"
Oct 30 11:35:40.237132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.237075    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tncl"
Oct 30 11:35:40.245019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.244987    5060 generic.go:356] "Generic (PLEG): container finished" podID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerID="fa6282d5aa9064a123df9fd07538451d5f9bf18f7ea31a2937acc46384222b12" exitCode=0
Oct 30 11:35:40.245487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.245436    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl" event={"ID":"6093080c-661e-48b3-b549-cd187ebc9ffc","Type":"ContainerDied","Data":"fa6282d5aa9064a123df9fd07538451d5f9bf18f7ea31a2937acc46384222b12"}
Oct 30 11:35:40.280826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.280763    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" podStartSLOduration=2.215909388 podStartE2EDuration="37.280742882s" totalImagesPullingTime="35.064833494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:40.2771321 +0000 UTC m=+344.072016622" watchObservedRunningTime="2026-10-30 11:35:40.280742882 +0000 UTC m=+344.075627377"
Oct 30 11:35:40.443783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.443758    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:40.544481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.544396    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p\" (UniqueName: \"kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p\") pod \"287868b1-74f6-4f01-9949-c58ce8078a57\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") "
Oct 30 11:35:40.544685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.544645    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle\") pod \"287868b1-74f6-4f01-9949-c58ce8078a57\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") "
Oct 30 11:35:40.544786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.544748    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util\") pod \"287868b1-74f6-4f01-9949-c58ce8078a57\" (UID: \"287868b1-74f6-4f01-9949-c58ce8078a57\") "
Oct 30 11:35:40.545531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.545506    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle" pod "287868b1-74f6-4f01-9949-c58ce8078a57" (UID: "287868b1-74f6-4f01-9949-c58ce8078a57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:40.550023 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.549751    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p" pod "287868b1-74f6-4f01-9949-c58ce8078a57" (UID: "287868b1-74f6-4f01-9949-c58ce8078a57"). InnerVolumeSpecName "kube-api-access-rhc5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:40.554524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.554497    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util" pod "287868b1-74f6-4f01-9949-c58ce8078a57" (UID: "287868b1-74f6-4f01-9949-c58ce8078a57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:40.645961 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.645931    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:40.645961 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.645959    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhc5p\" (UniqueName: \"kubernetes.io/projected/287868b1-74f6-4f01-9949-c58ce8078a57-kube-api-access-rhc5p\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:40.646128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:40.645969    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287868b1-74f6-4f01-9949-c58ce8078a57-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:41.252263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.252229    5060 generic.go:356] "Generic (PLEG): container finished" podID="dac7c339-9cca-4655-ab42-1d1a13615408" containerID="744c3f4459c0e12c1f6561678215a8e2236ae52b7c5bc2293cbe552a4783d452" exitCode=0
Oct 30 11:35:41.252740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.252312    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k" event={"ID":"dac7c339-9cca-4655-ab42-1d1a13615408","Type":"ContainerDied","Data":"744c3f4459c0e12c1f6561678215a8e2236ae52b7c5bc2293cbe552a4783d452"}
Oct 30 11:35:41.254226 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.254204    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc"
Oct 30 11:35:41.254226 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.254211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crzxqc" event={"ID":"287868b1-74f6-4f01-9949-c58ce8078a57","Type":"ContainerDied","Data":"acd8bb84a64246f3467708110f9066c74df49a4c1a067207dc923450fa9a3609"}
Oct 30 11:35:41.254391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.254249    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd8bb84a64246f3467708110f9066c74df49a4c1a067207dc923450fa9a3609"
Oct 30 11:35:41.256411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.256386    5060 generic.go:356] "Generic (PLEG): container finished" podID="d8bff443-e933-420d-b91a-7d537c54b376" containerID="567029ce27499150ecc8c076fd5ab7f5defc8f264a98d8a14f0618030ba3a972" exitCode=0
Oct 30 11:35:41.256534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.256475    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz" event={"ID":"d8bff443-e933-420d-b91a-7d537c54b376","Type":"ContainerDied","Data":"567029ce27499150ecc8c076fd5ab7f5defc8f264a98d8a14f0618030ba3a972"}
Oct 30 11:35:41.258511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.258489    5060 generic.go:356] "Generic (PLEG): container finished" podID="fdb1da02-1374-418b-9288-314da3ff59c7" containerID="618782e5e5a401efc65196057e34ca7861bd8a3418725be43a909dce2188470f" exitCode=0
Oct 30 11:35:41.258579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.258558    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf" event={"ID":"fdb1da02-1374-418b-9288-314da3ff59c7","Type":"ContainerDied","Data":"618782e5e5a401efc65196057e34ca7861bd8a3418725be43a909dce2188470f"}
Oct 30 11:35:41.260445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.260427    5060 generic.go:356] "Generic (PLEG): container finished" podID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerID="6e79525241735fd92e97189b3a239923e6b748fdbd0b075f5936a52a083833bb" exitCode=0
Oct 30 11:35:41.260541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.260489    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" event={"ID":"d34bd99c-7bd9-4aea-9969-840ff39ee70c","Type":"ContainerDied","Data":"6e79525241735fd92e97189b3a239923e6b748fdbd0b075f5936a52a083833bb"}
Oct 30 11:35:41.262628 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.262608    5060 generic.go:356] "Generic (PLEG): container finished" podID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerID="6164d12598b7173d2855f4615c783c2a20ffb512a381e6e257eccc12ebd783bc" exitCode=0
Oct 30 11:35:41.262719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.262674    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84" event={"ID":"1b94ea0c-e42e-45b6-b99d-a019b4bade19","Type":"ContainerDied","Data":"6164d12598b7173d2855f4615c783c2a20ffb512a381e6e257eccc12ebd783bc"}
Oct 30 11:35:41.264857 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.264839    5060 generic.go:356] "Generic (PLEG): container finished" podID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerID="774037e23fb32c7a04467b31cc6685773241599f8d0758c19b0843fc69d563f4" exitCode=0
Oct 30 11:35:41.264957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.264898    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb" event={"ID":"730f5ac7-70ed-4af4-b93b-126af23f43ff","Type":"ContainerDied","Data":"774037e23fb32c7a04467b31cc6685773241599f8d0758c19b0843fc69d563f4"}
Oct 30 11:35:41.378858 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.378834    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:41.451194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.451163    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l\" (UniqueName: \"kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l\") pod \"6093080c-661e-48b3-b549-cd187ebc9ffc\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") "
Oct 30 11:35:41.451328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.451237    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util\") pod \"6093080c-661e-48b3-b549-cd187ebc9ffc\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") "
Oct 30 11:35:41.451381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.451325    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle\") pod \"6093080c-661e-48b3-b549-cd187ebc9ffc\" (UID: \"6093080c-661e-48b3-b549-cd187ebc9ffc\") "
Oct 30 11:35:41.451746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.451719    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle" pod "6093080c-661e-48b3-b549-cd187ebc9ffc" (UID: "6093080c-661e-48b3-b549-cd187ebc9ffc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:41.453491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.453464    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l" pod "6093080c-661e-48b3-b549-cd187ebc9ffc" (UID: "6093080c-661e-48b3-b549-cd187ebc9ffc"). InnerVolumeSpecName "kube-api-access-hhs6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:41.456223 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.456205    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util" pod "6093080c-661e-48b3-b549-cd187ebc9ffc" (UID: "6093080c-661e-48b3-b549-cd187ebc9ffc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:41.553001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.552916    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:41.553001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.552944    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhs6l\" (UniqueName: \"kubernetes.io/projected/6093080c-661e-48b3-b549-cd187ebc9ffc-kube-api-access-hhs6l\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:41.553001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:41.552957    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6093080c-661e-48b3-b549-cd187ebc9ffc-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.271403 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.271369    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl"
Oct 30 11:35:42.271979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.271403    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6np9vl" event={"ID":"6093080c-661e-48b3-b549-cd187ebc9ffc","Type":"ContainerDied","Data":"47bafab79d720e8c675604b73f4a71c7317bbfddc7abf1240aa28a6650be9faf"}
Oct 30 11:35:42.271979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.271485    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47bafab79d720e8c675604b73f4a71c7317bbfddc7abf1240aa28a6650be9faf"
Oct 30 11:35:42.453416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.453355    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:42.506572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.503863    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:42.527537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.523981    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:42.529025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.529002    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:42.533181 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.533162    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:42.543228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.543204    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:42.563409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563377    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm\" (UniqueName: \"kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm\") pod \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") "
Oct 30 11:35:42.563605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563415    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util\") pod \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") "
Oct 30 11:35:42.563605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563489    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle\") pod \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\" (UID: \"1b94ea0c-e42e-45b6-b99d-a019b4bade19\") "
Oct 30 11:35:42.563605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563544    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle\") pod \"d8bff443-e933-420d-b91a-7d537c54b376\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") "
Oct 30 11:35:42.563605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr\" (UniqueName: \"kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr\") pod \"d8bff443-e933-420d-b91a-7d537c54b376\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") "
Oct 30 11:35:42.563887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util\") pod \"d8bff443-e933-420d-b91a-7d537c54b376\" (UID: \"d8bff443-e933-420d-b91a-7d537c54b376\") "
Oct 30 11:35:42.564016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.563881    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle" pod "1b94ea0c-e42e-45b6-b99d-a019b4bade19" (UID: "1b94ea0c-e42e-45b6-b99d-a019b4bade19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.565087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.565055    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle" pod "d8bff443-e933-420d-b91a-7d537c54b376" (UID: "d8bff443-e933-420d-b91a-7d537c54b376"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.565871 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.565844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm" pod "1b94ea0c-e42e-45b6-b99d-a019b4bade19" (UID: "1b94ea0c-e42e-45b6-b99d-a019b4bade19"). InnerVolumeSpecName "kube-api-access-d9jhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:42.566509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.566487    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr" pod "d8bff443-e933-420d-b91a-7d537c54b376" (UID: "d8bff443-e933-420d-b91a-7d537c54b376"). InnerVolumeSpecName "kube-api-access-hgdmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:42.569842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.569822    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util" pod "d8bff443-e933-420d-b91a-7d537c54b376" (UID: "d8bff443-e933-420d-b91a-7d537c54b376"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.570338 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.570317    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util" pod "1b94ea0c-e42e-45b6-b99d-a019b4bade19" (UID: "1b94ea0c-e42e-45b6-b99d-a019b4bade19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.664755 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.664727    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh\" (UniqueName: \"kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh\") pod \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") "
Oct 30 11:35:42.664755 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.664761    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util\") pod \"fdb1da02-1374-418b-9288-314da3ff59c7\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") "
Oct 30 11:35:42.665004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.664780    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle\") pod \"dac7c339-9cca-4655-ab42-1d1a13615408\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") "
Oct 30 11:35:42.665004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.664833    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr\" (UniqueName: \"kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr\") pod \"730f5ac7-70ed-4af4-b93b-126af23f43ff\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") "
Oct 30 11:35:42.665004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.664878    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util\") pod \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") "
Oct 30 11:35:42.665004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.664914    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util\") pod \"730f5ac7-70ed-4af4-b93b-126af23f43ff\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") "
Oct 30 11:35:42.665200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665030    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle\") pod \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\" (UID: \"d34bd99c-7bd9-4aea-9969-840ff39ee70c\") "
Oct 30 11:35:42.665200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665074    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle\") pod \"730f5ac7-70ed-4af4-b93b-126af23f43ff\" (UID: \"730f5ac7-70ed-4af4-b93b-126af23f43ff\") "
Oct 30 11:35:42.665200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665120    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util\") pod \"dac7c339-9cca-4655-ab42-1d1a13615408\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") "
Oct 30 11:35:42.665200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle\") pod \"fdb1da02-1374-418b-9288-314da3ff59c7\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") "
Oct 30 11:35:42.665439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665202    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d\" (UniqueName: \"kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d\") pod \"fdb1da02-1374-418b-9288-314da3ff59c7\" (UID: \"fdb1da02-1374-418b-9288-314da3ff59c7\") "
Oct 30 11:35:42.665439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665240    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp\" (UniqueName: \"kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp\") pod \"dac7c339-9cca-4655-ab42-1d1a13615408\" (UID: \"dac7c339-9cca-4655-ab42-1d1a13615408\") "
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665521    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle" pod "dac7c339-9cca-4655-ab42-1d1a13615408" (UID: "dac7c339-9cca-4655-ab42-1d1a13615408"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665546    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jhm\" (UniqueName: \"kubernetes.io/projected/1b94ea0c-e42e-45b6-b99d-a019b4bade19-kube-api-access-d9jhm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665566    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665580    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b94ea0c-e42e-45b6-b99d-a019b4bade19-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665592    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665605    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgdmr\" (UniqueName: \"kubernetes.io/projected/d8bff443-e933-420d-b91a-7d537c54b376-kube-api-access-hgdmr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.665673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.665618    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8bff443-e933-420d-b91a-7d537c54b376-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.666087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.666046    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle" pod "d34bd99c-7bd9-4aea-9969-840ff39ee70c" (UID: "d34bd99c-7bd9-4aea-9969-840ff39ee70c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.666708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.666680    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle" pod "fdb1da02-1374-418b-9288-314da3ff59c7" (UID: "fdb1da02-1374-418b-9288-314da3ff59c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.667404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.667368    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle" pod "730f5ac7-70ed-4af4-b93b-126af23f43ff" (UID: "730f5ac7-70ed-4af4-b93b-126af23f43ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.667654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.667624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh" pod "d34bd99c-7bd9-4aea-9969-840ff39ee70c" (UID: "d34bd99c-7bd9-4aea-9969-840ff39ee70c"). InnerVolumeSpecName "kube-api-access-kgwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:42.668108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.668083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp" pod "dac7c339-9cca-4655-ab42-1d1a13615408" (UID: "dac7c339-9cca-4655-ab42-1d1a13615408"). InnerVolumeSpecName "kube-api-access-g6nhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:42.669873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.669849    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d" pod "fdb1da02-1374-418b-9288-314da3ff59c7" (UID: "fdb1da02-1374-418b-9288-314da3ff59c7"). InnerVolumeSpecName "kube-api-access-qkc8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:42.670056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.669994    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr" pod "730f5ac7-70ed-4af4-b93b-126af23f43ff" (UID: "730f5ac7-70ed-4af4-b93b-126af23f43ff"). InnerVolumeSpecName "kube-api-access-442vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:35:42.673257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.673232    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util" pod "730f5ac7-70ed-4af4-b93b-126af23f43ff" (UID: "730f5ac7-70ed-4af4-b93b-126af23f43ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.673539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.673513    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util" pod "d34bd99c-7bd9-4aea-9969-840ff39ee70c" (UID: "d34bd99c-7bd9-4aea-9969-840ff39ee70c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.674332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.674312    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util" pod "fdb1da02-1374-418b-9288-314da3ff59c7" (UID: "fdb1da02-1374-418b-9288-314da3ff59c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.675227 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.675202    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util" pod "dac7c339-9cca-4655-ab42-1d1a13615408" (UID: "dac7c339-9cca-4655-ab42-1d1a13615408"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:35:42.766801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766771    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.766801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766797    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.766801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766806    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/730f5ac7-70ed-4af4-b93b-126af23f43ff-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766813    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766822    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766830    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkc8d\" (UniqueName: \"kubernetes.io/projected/fdb1da02-1374-418b-9288-314da3ff59c7-kube-api-access-qkc8d\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766840    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6nhp\" (UniqueName: \"kubernetes.io/projected/dac7c339-9cca-4655-ab42-1d1a13615408-kube-api-access-g6nhp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766848    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgwkh\" (UniqueName: \"kubernetes.io/projected/d34bd99c-7bd9-4aea-9969-840ff39ee70c-kube-api-access-kgwkh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766856    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdb1da02-1374-418b-9288-314da3ff59c7-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766864    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac7c339-9cca-4655-ab42-1d1a13615408-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766872    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-442vr\" (UniqueName: \"kubernetes.io/projected/730f5ac7-70ed-4af4-b93b-126af23f43ff-kube-api-access-442vr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:42.767013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:42.766879    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34bd99c-7bd9-4aea-9969-840ff39ee70c-util\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:35:43.277435 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.277393    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84" event={"ID":"1b94ea0c-e42e-45b6-b99d-a019b4bade19","Type":"ContainerDied","Data":"575f3169cc5848690efa2c68e72a8efc498718983956269f20c34523ccc27000"}
Oct 30 11:35:43.277911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.277409    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gp84"
Oct 30 11:35:43.277911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.277483    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575f3169cc5848690efa2c68e72a8efc498718983956269f20c34523ccc27000"
Oct 30 11:35:43.279490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.279443    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb"
Oct 30 11:35:43.279610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.279474    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tptxb" event={"ID":"730f5ac7-70ed-4af4-b93b-126af23f43ff","Type":"ContainerDied","Data":"8488976a2b1255cc8757a94b2693f0851a18f64e723fc9311c943abe38ab119a"}
Oct 30 11:35:43.279610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.279602    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8488976a2b1255cc8757a94b2693f0851a18f64e723fc9311c943abe38ab119a"
Oct 30 11:35:43.281289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.281271    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k"
Oct 30 11:35:43.281427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.281288    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tld9k" event={"ID":"dac7c339-9cca-4655-ab42-1d1a13615408","Type":"ContainerDied","Data":"3700caa7b2504d1af9a8050a4140974f90a4f7ff89c536d4f6b169dce86f1054"}
Oct 30 11:35:43.281427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.281324    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3700caa7b2504d1af9a8050a4140974f90a4f7ff89c536d4f6b169dce86f1054"
Oct 30 11:35:43.283312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.283292    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz" event={"ID":"d8bff443-e933-420d-b91a-7d537c54b376","Type":"ContainerDied","Data":"145cddd4569c0c0aeacdc87719eb98265ee68bc9a33a66a992c95b86e76c76f8"}
Oct 30 11:35:43.283436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.283326    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="145cddd4569c0c0aeacdc87719eb98265ee68bc9a33a66a992c95b86e76c76f8"
Oct 30 11:35:43.283436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.283296    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8bjlz"
Oct 30 11:35:43.285421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.285397    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf" event={"ID":"fdb1da02-1374-418b-9288-314da3ff59c7","Type":"ContainerDied","Data":"75d62a8caef59c38442793e38f182d0ca9853dbd7d2d04cc50afe37b30015d8e"}
Oct 30 11:35:43.285540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.285438    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d62a8caef59c38442793e38f182d0ca9853dbd7d2d04cc50afe37b30015d8e"
Oct 30 11:35:43.285540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.285404    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9wkcf"
Oct 30 11:35:43.287700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.287682    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5"
Oct 30 11:35:43.287823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.287676    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg6hk5" event={"ID":"d34bd99c-7bd9-4aea-9969-840ff39ee70c","Type":"ContainerDied","Data":"758318319cf85b6e318c1d6312d2b5c8668360c0962199eeed39f1f703726722"}
Oct 30 11:35:43.287823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.287798    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758318319cf85b6e318c1d6312d2b5c8668360c0962199eeed39f1f703726722"
Oct 30 11:35:43.625693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.625609    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:43.786022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:43.785989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:44.772736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.772702    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"]
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773027    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="util"
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773039    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="util"
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773087    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="extract"
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773093    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="extract"
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773108    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="extract"
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773115    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="extract"
Oct 30 11:35:44.773124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773127    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773132    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773139    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773144    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773150    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773157    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773169    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773175    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773190    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773194    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773199    5060 cpu_manager.go:414] "containerMap: removing container" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773204    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773211    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773215    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773221    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773225    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773233    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773240    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773252    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773258    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773264    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773268    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773274    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773279    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773286    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773292    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773299    5060 cpu_manager.go:414] "containerMap: removing container" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773303    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="util"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773309    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773313    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="pull"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773322    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773329    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773338    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="extract"
Oct 30 11:35:44.773332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773343    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773351    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="util"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773355    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="util"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773361    5060 cpu_manager.go:414] "containerMap: removing container" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773365    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773370    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="util"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773374    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="util"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773379    5060 cpu_manager.go:414] "containerMap: removing container" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="util"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773383    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="util"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773389    5060 cpu_manager.go:414] "containerMap: removing container" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773394    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773401    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773409    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773418    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773422    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773427    5060 cpu_manager.go:414] "containerMap: removing container" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773431    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773439    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773444    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773462    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773466    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="pull"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773531    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb1da02-1374-418b-9288-314da3ff59c7" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773538    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f647cd7-39c3-4094-8439-3d221fbc4eec" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773545    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="287868b1-74f6-4f01-9949-c58ce8078a57" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773552    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b11fdfd-8734-44f2-ab59-eb247f02c155" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773558    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34bd99c-7bd9-4aea-9969-840ff39ee70c" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773563    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b94ea0c-e42e-45b6-b99d-a019b4bade19" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773569    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac7c339-9cca-4655-ab42-1d1a13615408" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773574    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="730f5ac7-70ed-4af4-b93b-126af23f43ff" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773579    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6093080c-661e-48b3-b549-cd187ebc9ffc" containerName="extract"
Oct 30 11:35:44.774275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.773585    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bff443-e933-420d-b91a-7d537c54b376" containerName="extract"
Oct 30 11:35:44.778046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.778027    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:44.795577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.795556    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 11:35:44.807031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.807008    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 11:35:44.807148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.807059    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jgndz\""
Oct 30 11:35:44.808844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.808818    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"]
Oct 30 11:35:44.881856 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.881817    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bwb\" (UniqueName: \"kubernetes.io/projected/8d63a619-badf-4a88-b808-365824fc423b-kube-api-access-c9bwb\") pod \"cert-manager-webhook-5dbbbf75cc-bfswl\" (UID: \"8d63a619-badf-4a88-b808-365824fc423b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:44.882008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.881866    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d63a619-badf-4a88-b808-365824fc423b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bfswl\" (UID: \"8d63a619-badf-4a88-b808-365824fc423b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:44.982902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.982862    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bwb\" (UniqueName: \"kubernetes.io/projected/8d63a619-badf-4a88-b808-365824fc423b-kube-api-access-c9bwb\") pod \"cert-manager-webhook-5dbbbf75cc-bfswl\" (UID: \"8d63a619-badf-4a88-b808-365824fc423b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:44.983067 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:44.982921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d63a619-badf-4a88-b808-365824fc423b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bfswl\" (UID: \"8d63a619-badf-4a88-b808-365824fc423b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:45.007543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:45.007510    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d63a619-badf-4a88-b808-365824fc423b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bfswl\" (UID: \"8d63a619-badf-4a88-b808-365824fc423b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:45.007805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:45.007784    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bwb\" (UniqueName: \"kubernetes.io/projected/8d63a619-badf-4a88-b808-365824fc423b-kube-api-access-c9bwb\") pod \"cert-manager-webhook-5dbbbf75cc-bfswl\" (UID: \"8d63a619-badf-4a88-b808-365824fc423b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:45.099975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:45.099891    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:45.395893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:45.395867    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"]
Oct 30 11:35:45.396539 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:45.396513    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d63a619_badf_4a88_b808_365824fc423b.slice/crio-654953d95aa4bfc982ae882044f1032e82c5de7128c7d3f71a4996c87a96b3f5 WatchSource:0}: Error finding container 654953d95aa4bfc982ae882044f1032e82c5de7128c7d3f71a4996c87a96b3f5: Status 404 returned error can't find the container with id 654953d95aa4bfc982ae882044f1032e82c5de7128c7d3f71a4996c87a96b3f5
Oct 30 11:35:46.303250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:46.303199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl" event={"ID":"8d63a619-badf-4a88-b808-365824fc423b","Type":"ContainerStarted","Data":"654953d95aa4bfc982ae882044f1032e82c5de7128c7d3f71a4996c87a96b3f5"}
Oct 30 11:35:46.747465 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:46.747417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:47.623840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:47.623801    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:47.853942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:47.853901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:48.032296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.032261    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:48.529669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.529637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:48.670796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.670758    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"]
Oct 30 11:35:48.673629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.673565    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.691313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.691289    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 11:35:48.691577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.691562    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 11:35:48.703895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.703873    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 11:35:48.703895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.703874    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 11:35:48.704033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.703911    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bjbjc\""
Oct 30 11:35:48.704033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.703955    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 11:35:48.710408 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.710384    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"]
Oct 30 11:35:48.824531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.824496    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6c7ad34-729f-494d-9ed3-7b438ec13e05-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.824722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.824546    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.824722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.824673    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.824722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.824702    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpf7\" (UniqueName: \"kubernetes.io/projected/f6c7ad34-729f-494d-9ed3-7b438ec13e05-kube-api-access-8rpf7\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.824827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.824740    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6c7ad34-729f-494d-9ed3-7b438ec13e05-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.824860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.824816    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.891500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.891444    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:48.925572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.925509    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6c7ad34-729f-494d-9ed3-7b438ec13e05-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.925572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.925547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.925736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.925580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6c7ad34-729f-494d-9ed3-7b438ec13e05-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.925736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.925618    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.925736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.925654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.925736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.925687    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpf7\" (UniqueName: \"kubernetes.io/projected/f6c7ad34-729f-494d-9ed3-7b438ec13e05-kube-api-access-8rpf7\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.926089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.926054    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6c7ad34-729f-494d-9ed3-7b438ec13e05-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.928219 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.928183    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.928312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.928239    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.928392 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.928377    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6c7ad34-729f-494d-9ed3-7b438ec13e05-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.928605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.928590    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c7ad34-729f-494d-9ed3-7b438ec13e05-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.955858 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.955833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpf7\" (UniqueName: \"kubernetes.io/projected/f6c7ad34-729f-494d-9ed3-7b438ec13e05-kube-api-access-8rpf7\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-p5687\" (UID: \"f6c7ad34-729f-494d-9ed3-7b438ec13e05\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:48.986677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:48.986648    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:49.089438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.089406    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:49.288119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.288084    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:49.320636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.320603    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl" event={"ID":"8d63a619-badf-4a88-b808-365824fc423b","Type":"ContainerStarted","Data":"f683e59e9110b1ecba870cf3136dcffbbec1057178891f44777a179f1611b0f0"}
Oct 30 11:35:49.320827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.320649    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:49.345295 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:49.345270    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c7ad34_729f_494d_9ed3_7b438ec13e05.slice/crio-d429e6a1566bb6b96dc69247ff5fbcdbd55c28f32d202ade83b6dd814904805e WatchSource:0}: Error finding container d429e6a1566bb6b96dc69247ff5fbcdbd55c28f32d202ade83b6dd814904805e: Status 404 returned error can't find the container with id d429e6a1566bb6b96dc69247ff5fbcdbd55c28f32d202ade83b6dd814904805e
Oct 30 11:35:49.346183 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.346160    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"]
Oct 30 11:35:49.366964 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.366918    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl" podStartSLOduration=2.343399674 podStartE2EDuration="5.366904149s" totalImagesPullingTime="3.023504475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:49.366642727 +0000 UTC m=+353.161527221" watchObservedRunningTime="2026-10-30 11:35:49.366904149 +0000 UTC m=+353.161788642"
Oct 30 11:35:49.454810 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.454777    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:49.626989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:49.626901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:49.855569 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:35:49.855532    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:35:50.338753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:50.338710    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687" event={"ID":"f6c7ad34-729f-494d-9ed3-7b438ec13e05","Type":"ContainerStarted","Data":"d429e6a1566bb6b96dc69247ff5fbcdbd55c28f32d202ade83b6dd814904805e"}
Oct 30 11:35:51.972272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:51.972232    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:52.170229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:52.170192    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:52.339804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:52.339770    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687" event={"ID":"f6c7ad34-729f-494d-9ed3-7b438ec13e05","Type":"ContainerStarted","Data":"4bfe175e6c075630d501ac88c5be158b650c0996543f18c5da9119d39fa00a31"}
Oct 30 11:35:53.345987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:53.345953    5060 generic.go:356] "Generic (PLEG): container finished" podID="f6c7ad34-729f-494d-9ed3-7b438ec13e05" containerID="4bfe175e6c075630d501ac88c5be158b650c0996543f18c5da9119d39fa00a31" exitCode=0
Oct 30 11:35:53.346390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:53.346033    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687" event={"ID":"f6c7ad34-729f-494d-9ed3-7b438ec13e05","Type":"ContainerDied","Data":"4bfe175e6c075630d501ac88c5be158b650c0996543f18c5da9119d39fa00a31"}
Oct 30 11:35:54.246375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:54.246335    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:54.352930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:54.352895    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687" event={"ID":"f6c7ad34-729f-494d-9ed3-7b438ec13e05","Type":"ContainerStarted","Data":"29c44920450c7cfbeaa84963b00afbb435c76d4b55388b9eec7bef59a643ec09"}
Oct 30 11:35:54.353352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:54.353002    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:35:54.445414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:54.445361    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687" podStartSLOduration=2.851039061 podStartE2EDuration="6.445347396s" totalImagesPullingTime="3.176092447s" totalInitContainerRuntime="418.215888ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:54.432203643 +0000 UTC m=+358.227088147" watchObservedRunningTime="2026-10-30 11:35:54.445347396 +0000 UTC m=+358.240231889"
Oct 30 11:35:55.330013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:55.329979    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bfswl"
Oct 30 11:35:57.012791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.012758    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"]
Oct 30 11:35:57.017597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.017582    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:35:57.048250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.048208    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 11:35:57.048250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.048219    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 11:35:57.048250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.048243    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 11:35:57.048499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.048393    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qhf4v\""
Oct 30 11:35:57.051642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.051624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"]
Oct 30 11:35:57.199025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.198990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscn7\" (UniqueName: \"kubernetes.io/projected/d621ee63-03be-454b-b1b1-9e54a635433b-kube-api-access-xscn7\") pod \"dns-operator-controller-manager-8557cbdb84-lfxxm\" (UID: \"d621ee63-03be-454b-b1b1-9e54a635433b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:35:57.300277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.300182    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscn7\" (UniqueName: \"kubernetes.io/projected/d621ee63-03be-454b-b1b1-9e54a635433b-kube-api-access-xscn7\") pod \"dns-operator-controller-manager-8557cbdb84-lfxxm\" (UID: \"d621ee63-03be-454b-b1b1-9e54a635433b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:35:57.324336 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.324304    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscn7\" (UniqueName: \"kubernetes.io/projected/d621ee63-03be-454b-b1b1-9e54a635433b-kube-api-access-xscn7\") pod \"dns-operator-controller-manager-8557cbdb84-lfxxm\" (UID: \"d621ee63-03be-454b-b1b1-9e54a635433b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:35:57.329395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.329370    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:35:57.627747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:35:57.627716    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd621ee63_03be_454b_b1b1_9e54a635433b.slice/crio-c45f176f221b88255ffc0ac68d3fc3f4cabc2db8e01c4ececc5030b7b0892801 WatchSource:0}: Error finding container c45f176f221b88255ffc0ac68d3fc3f4cabc2db8e01c4ececc5030b7b0892801: Status 404 returned error can't find the container with id c45f176f221b88255ffc0ac68d3fc3f4cabc2db8e01c4ececc5030b7b0892801
Oct 30 11:35:57.628108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.628087    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"]
Oct 30 11:35:57.812200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:57.812163    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:58.377901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:58.377869    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm" event={"ID":"d621ee63-03be-454b-b1b1-9e54a635433b","Type":"ContainerStarted","Data":"c45f176f221b88255ffc0ac68d3fc3f4cabc2db8e01c4ececc5030b7b0892801"}
Oct 30 11:35:58.415068 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:58.415035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:35:59.018539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:35:59.018500    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:00.050844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:00.050805    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:00.391161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:00.391082    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm" event={"ID":"d621ee63-03be-454b-b1b1-9e54a635433b","Type":"ContainerStarted","Data":"22f90ef1da1d0433f52a25547cf39da3d24f99fd5af6dd51944c299a97a63430"}
Oct 30 11:36:00.391310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:00.391221    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:36:00.434793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:00.434740    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm" podStartSLOduration=1.836412586 podStartE2EDuration="4.434726798s" totalImagesPullingTime="2.598314212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:00.433633127 +0000 UTC m=+364.228517633" watchObservedRunningTime="2026-10-30 11:36:00.434726798 +0000 UTC m=+364.229611292"
Oct 30 11:36:02.075864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.075833    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:02.250138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.250107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:02.530677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.530643    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"]
Oct 30 11:36:02.534140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.534124    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.544831 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.544804    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1312742-65e5-4fd2-9751-0a260930048f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7h64q\" (UID: \"e1312742-65e5-4fd2-9751-0a260930048f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.544954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.544866    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdrp\" (UniqueName: \"kubernetes.io/projected/e1312742-65e5-4fd2-9751-0a260930048f-kube-api-access-scdrp\") pod \"kuadrant-operator-controller-manager-678d87766f-7h64q\" (UID: \"e1312742-65e5-4fd2-9751-0a260930048f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.546433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.546412    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"]
Oct 30 11:36:02.556757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.556740    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j2kz8\""
Oct 30 11:36:02.645255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.645233    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdrp\" (UniqueName: \"kubernetes.io/projected/e1312742-65e5-4fd2-9751-0a260930048f-kube-api-access-scdrp\") pod \"kuadrant-operator-controller-manager-678d87766f-7h64q\" (UID: \"e1312742-65e5-4fd2-9751-0a260930048f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.645367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.645276    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1312742-65e5-4fd2-9751-0a260930048f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7h64q\" (UID: \"e1312742-65e5-4fd2-9751-0a260930048f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.645631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.645614    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1312742-65e5-4fd2-9751-0a260930048f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7h64q\" (UID: \"e1312742-65e5-4fd2-9751-0a260930048f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.670979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.670957    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdrp\" (UniqueName: \"kubernetes.io/projected/e1312742-65e5-4fd2-9751-0a260930048f-kube-api-access-scdrp\") pod \"kuadrant-operator-controller-manager-678d87766f-7h64q\" (UID: \"e1312742-65e5-4fd2-9751-0a260930048f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:02.847048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:02.846964    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:03.020116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:03.020079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:03.178267 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:03.178188    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"]
Oct 30 11:36:03.190304 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:36:03.190277    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1312742_65e5_4fd2_9751_0a260930048f.slice/crio-fb09b2a8f123bc62968e42ebedce3b7552ee288533456cfa722372575750b0aa WatchSource:0}: Error finding container fb09b2a8f123bc62968e42ebedce3b7552ee288533456cfa722372575750b0aa: Status 404 returned error can't find the container with id fb09b2a8f123bc62968e42ebedce3b7552ee288533456cfa722372575750b0aa
Oct 30 11:36:03.406520 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:03.406487    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q" event={"ID":"e1312742-65e5-4fd2-9751-0a260930048f","Type":"ContainerStarted","Data":"fb09b2a8f123bc62968e42ebedce3b7552ee288533456cfa722372575750b0aa"}
Oct 30 11:36:04.610004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.609961    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:04.751110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.751077    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"]
Oct 30 11:36:04.754660 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.754637    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:04.760582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.760556    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7n9\" (UniqueName: \"kubernetes.io/projected/ba17fa44-3f35-497d-b2ce-b4a6415511dc-kube-api-access-9s7n9\") pod \"limitador-operator-controller-manager-5778bb4449-pzz7z\" (UID: \"ba17fa44-3f35-497d-b2ce-b4a6415511dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:04.786515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.786494    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5mn7d\""
Oct 30 11:36:04.788444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.788421    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"]
Oct 30 11:36:04.861088 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.861012    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7n9\" (UniqueName: \"kubernetes.io/projected/ba17fa44-3f35-497d-b2ce-b4a6415511dc-kube-api-access-9s7n9\") pod \"limitador-operator-controller-manager-5778bb4449-pzz7z\" (UID: \"ba17fa44-3f35-497d-b2ce-b4a6415511dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:04.910466 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.910409    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7n9\" (UniqueName: \"kubernetes.io/projected/ba17fa44-3f35-497d-b2ce-b4a6415511dc-kube-api-access-9s7n9\") pod \"limitador-operator-controller-manager-5778bb4449-pzz7z\" (UID: \"ba17fa44-3f35-497d-b2ce-b4a6415511dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:04.928040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:04.927963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:05.070388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:05.070344    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:05.154878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:05.154845    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:05.361495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:05.361447    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-p5687"
Oct 30 11:36:05.406886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:05.406855    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"]
Oct 30 11:36:05.420762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:05.420736    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z" event={"ID":"ba17fa44-3f35-497d-b2ce-b4a6415511dc","Type":"ContainerStarted","Data":"443369be6711dfab887edc0258026e12a045681dc6fccd2357d666dc4dd28cef"}
Oct 30 11:36:07.133395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.133363    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:07.331634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.331595    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:07.380667 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.380627    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"]
Oct 30 11:36:07.384225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.384202    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:07.400199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.400177    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x5fg8\""
Oct 30 11:36:07.414920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.414896    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"]
Oct 30 11:36:07.434068 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.434032    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q" event={"ID":"e1312742-65e5-4fd2-9751-0a260930048f","Type":"ContainerStarted","Data":"02d0ec669dae05c92af961230643ec6088d64bc046b7c719ba94f8bd5aeb7547"}
Oct 30 11:36:07.434270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.434128    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:07.437097 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.437074    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z" event={"ID":"ba17fa44-3f35-497d-b2ce-b4a6415511dc","Type":"ContainerStarted","Data":"30322f6363528b5cf9dc78d23d7523a453b47b1f417df9423d6c97445b086bed"}
Oct 30 11:36:07.437200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.437168    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:07.487527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.487491    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8gt\" (UniqueName: \"kubernetes.io/projected/c85b9afe-dc24-4cf9-8639-a59f727c4bb2-kube-api-access-mk8gt\") pod \"authorino-operator-7b7bb4bc77-h4nph\" (UID: \"c85b9afe-dc24-4cf9-8639-a59f727c4bb2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:07.501958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.501896    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z" podStartSLOduration=1.9781355980000002 podStartE2EDuration="3.501872771s" totalImagesPullingTime="1.523737173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:07.499726238 +0000 UTC m=+371.294610745" watchObservedRunningTime="2026-10-30 11:36:07.501872771 +0000 UTC m=+371.296757276"
Oct 30 11:36:07.549891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.549832    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q" podStartSLOduration=1.652060015 podStartE2EDuration="5.549816797s" totalImagesPullingTime="3.897756782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:07.547870968 +0000 UTC m=+371.342755481" watchObservedRunningTime="2026-10-30 11:36:07.549816797 +0000 UTC m=+371.344701292"
Oct 30 11:36:07.554117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.554085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:07.587999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.587921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8gt\" (UniqueName: \"kubernetes.io/projected/c85b9afe-dc24-4cf9-8639-a59f727c4bb2-kube-api-access-mk8gt\") pod \"authorino-operator-7b7bb4bc77-h4nph\" (UID: \"c85b9afe-dc24-4cf9-8639-a59f727c4bb2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:07.611932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.611896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8gt\" (UniqueName: \"kubernetes.io/projected/c85b9afe-dc24-4cf9-8639-a59f727c4bb2-kube-api-access-mk8gt\") pod \"authorino-operator-7b7bb4bc77-h4nph\" (UID: \"c85b9afe-dc24-4cf9-8639-a59f727c4bb2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:07.696743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.696711    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:07.716171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.716137    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:07.996511 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:36:07.996485    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85b9afe_dc24_4cf9_8639_a59f727c4bb2.slice/crio-065e980df1331ea9bb9a9d1173fcec51ff1ddd587ad9ebaf667b006506f93b35 WatchSource:0}: Error finding container 065e980df1331ea9bb9a9d1173fcec51ff1ddd587ad9ebaf667b006506f93b35: Status 404 returned error can't find the container with id 065e980df1331ea9bb9a9d1173fcec51ff1ddd587ad9ebaf667b006506f93b35
Oct 30 11:36:07.996953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:07.996932    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"]
Oct 30 11:36:08.444189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:08.444100    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph" event={"ID":"c85b9afe-dc24-4cf9-8639-a59f727c4bb2","Type":"ContainerStarted","Data":"065e980df1331ea9bb9a9d1173fcec51ff1ddd587ad9ebaf667b006506f93b35"}
Oct 30 11:36:08.666149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:08.666113    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:08.881621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:08.881581    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:09.130213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:09.130063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:09.299203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:09.299170    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:10.062602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:10.062559    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:10.237374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:10.237332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:10.377063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:10.376973    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:10.457716 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:10.457685    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph" event={"ID":"c85b9afe-dc24-4cf9-8639-a59f727c4bb2","Type":"ContainerStarted","Data":"9c5555ec6b0a6a863563a651566c4578a9baa32a8ebc78f8501cbd1b6bead830"}
Oct 30 11:36:10.457868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:10.457832    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:10.501266 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:10.501209    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph" podStartSLOduration=2.111024981 podStartE2EDuration="3.501193123s" totalImagesPullingTime="1.390168142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:10.499282871 +0000 UTC m=+374.294167359" watchObservedRunningTime="2026-10-30 11:36:10.501193123 +0000 UTC m=+374.296077613"
Oct 30 11:36:11.398506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:11.398473    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lfxxm"
Oct 30 11:36:13.409751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:13.409713    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:13.563204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:13.563171    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:13.711538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:13.711504    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:13.881094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:13.881060    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:18.446240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:18.446208    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pzz7z"
Oct 30 11:36:18.446787 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:18.446281    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7h64q"
Oct 30 11:36:19.856874 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:36:19.856834    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:36:20.392678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:20.392630    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:20.548719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:20.548679    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:21.467138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:21.467103    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 11:36:25.140641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:25.140608    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:25.288892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:25.288858    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:27.760579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:27.760540    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:27.927969 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:27.927935    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:28.187080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:28.186990    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:28.410596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:28.410567    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:28.777916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:28.777866    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:28.971553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:28.971513    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:29.075009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.074911    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"]
Oct 30 11:36:29.081428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.081399    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.102273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.102242    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"]
Oct 30 11:36:29.111910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.111877    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fnpfs\""
Oct 30 11:36:29.170973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.170926    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a346544-3796-45ea-81ee-cfba3265d767-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.170987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcz5\" (UniqueName: \"kubernetes.io/projected/6a346544-3796-45ea-81ee-cfba3265d767-kube-api-access-bjcz5\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171107    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a346544-3796-45ea-81ee-cfba3265d767-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171142    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171264 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171160    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a346544-3796-45ea-81ee-cfba3265d767-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171248    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171299    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.171435 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.171408    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.180575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.180545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:29.272681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272631    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.272681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272691    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.272941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272727    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.272941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.272941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a346544-3796-45ea-81ee-cfba3265d767-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.272941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272809    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcz5\" (UniqueName: \"kubernetes.io/projected/6a346544-3796-45ea-81ee-cfba3265d767-kube-api-access-bjcz5\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.272941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.272844    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a346544-3796-45ea-81ee-cfba3265d767-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273018    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a346544-3796-45ea-81ee-cfba3265d767-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273166    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273217    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.273725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.273709    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a346544-3796-45ea-81ee-cfba3265d767-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.275289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.275266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a346544-3796-45ea-81ee-cfba3265d767-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.275597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.275576    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a346544-3796-45ea-81ee-cfba3265d767-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.297192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.297150    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a346544-3796-45ea-81ee-cfba3265d767-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.297403 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.297380    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcz5\" (UniqueName: \"kubernetes.io/projected/6a346544-3796-45ea-81ee-cfba3265d767-kube-api-access-bjcz5\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft92xg\" (UID: \"6a346544-3796-45ea-81ee-cfba3265d767\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.343242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.343149    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:29.393680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.393638    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:29.488488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.488095    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:29.640692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.640596    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:29.728357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.725544    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"]
Oct 30 11:36:29.728969 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:36:29.728936    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a346544_3796_45ea_81ee_cfba3265d767.slice/crio-26328ad876b796021c8872b70ebb65961ad50e1aaca2ef9676442d7a5f1c3e23 WatchSource:0}: Error finding container 26328ad876b796021c8872b70ebb65961ad50e1aaca2ef9676442d7a5f1c3e23: Status 404 returned error can't find the container with id 26328ad876b796021c8872b70ebb65961ad50e1aaca2ef9676442d7a5f1c3e23
Oct 30 11:36:29.730910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.730882    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:36:29.730980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.730948    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:36:29.731016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:29.730982    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:36:30.078925 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:30.078885    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:30.229727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:30.229693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:30.572535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:30.572502    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg" event={"ID":"6a346544-3796-45ea-81ee-cfba3265d767","Type":"ContainerStarted","Data":"bf81dee7efdab4fea7deeb7ff3551e9da223c4505168b5b853c60b8dd8c31d43"}
Oct 30 11:36:30.572719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:30.572549    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg" event={"ID":"6a346544-3796-45ea-81ee-cfba3265d767","Type":"ContainerStarted","Data":"26328ad876b796021c8872b70ebb65961ad50e1aaca2ef9676442d7a5f1c3e23"}
Oct 30 11:36:30.639880 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:30.639821    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg" podStartSLOduration=1.639805021 podStartE2EDuration="1.639805021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:30.636597553 +0000 UTC m=+394.431482059" watchObservedRunningTime="2026-10-30 11:36:30.639805021 +0000 UTC m=+394.434689515"
Oct 30 11:36:30.866422 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:30.866336    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:31.126215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.126132    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x7t56"]
Oct 30 11:36:31.129282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.129263    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:31.147722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.147692    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7lq8x\""
Oct 30 11:36:31.148960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.148926    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x7t56"]
Oct 30 11:36:31.190119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.190079    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5skj\" (UniqueName: \"kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj\") pod \"authorino-57f7f8659f-x7t56\" (UID: \"8fda7bf5-994b-47a9-9146-8211c7aeb639\") " pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:31.276249 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.276201    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:31.291543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.291512    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5skj\" (UniqueName: \"kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj\") pod \"authorino-57f7f8659f-x7t56\" (UID: \"8fda7bf5-994b-47a9-9146-8211c7aeb639\") " pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:31.315878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.315849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5skj\" (UniqueName: \"kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj\") pod \"authorino-57f7f8659f-x7t56\" (UID: \"8fda7bf5-994b-47a9-9146-8211c7aeb639\") " pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:31.393984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.393907    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:31.398980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.398955    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:31.442899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.442862    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:31.524936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.524899    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:31.577766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.577734    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:31.578672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.578657    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft92xg"
Oct 30 11:36:31.714379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.714347    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bl7jf"]
Oct 30 11:36:31.718730 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.718711    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:36:31.755966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.755938    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bl7jf"]
Oct 30 11:36:31.771614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.771580    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x7t56"]
Oct 30 11:36:31.780575 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:36:31.780546    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fda7bf5_994b_47a9_9146_8211c7aeb639.slice/crio-6c2fdee439e6d03f663e814021ca1ec9daf23de640d990cfa1415b13e9a44b14 WatchSource:0}: Error finding container 6c2fdee439e6d03f663e814021ca1ec9daf23de640d990cfa1415b13e9a44b14: Status 404 returned error can't find the container with id 6c2fdee439e6d03f663e814021ca1ec9daf23de640d990cfa1415b13e9a44b14
Oct 30 11:36:31.795678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.795650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4vc\" (UniqueName: \"kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc\") pod \"authorino-56c6c9ddd5-bl7jf\" (UID: \"c645ca89-5a90-4a76-9daa-8a1a9818a807\") " pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:36:31.896687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.896642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4vc\" (UniqueName: \"kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc\") pod \"authorino-56c6c9ddd5-bl7jf\" (UID: \"c645ca89-5a90-4a76-9daa-8a1a9818a807\") " pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:36:31.926641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:31.921734    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4vc\" (UniqueName: \"kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc\") pod \"authorino-56c6c9ddd5-bl7jf\" (UID: \"c645ca89-5a90-4a76-9daa-8a1a9818a807\") " pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:36:32.030215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:32.030132    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:36:32.114328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:32.113213    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:32.350734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:32.350702    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bl7jf"]
Oct 30 11:36:32.351989 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:36:32.351956    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc645ca89_5a90_4a76_9daa_8a1a9818a807.slice/crio-5785f261c89ff5e1a87f96f1b6ca862fcc77cd57fc2dbebe1d4ec8830352c2c2 WatchSource:0}: Error finding container 5785f261c89ff5e1a87f96f1b6ca862fcc77cd57fc2dbebe1d4ec8830352c2c2: Status 404 returned error can't find the container with id 5785f261c89ff5e1a87f96f1b6ca862fcc77cd57fc2dbebe1d4ec8830352c2c2
Oct 30 11:36:32.584663 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:32.584620    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf" event={"ID":"c645ca89-5a90-4a76-9daa-8a1a9818a807","Type":"ContainerStarted","Data":"5785f261c89ff5e1a87f96f1b6ca862fcc77cd57fc2dbebe1d4ec8830352c2c2"}
Oct 30 11:36:32.586112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:32.586072    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x7t56" event={"ID":"8fda7bf5-994b-47a9-9146-8211c7aeb639","Type":"ContainerStarted","Data":"6c2fdee439e6d03f663e814021ca1ec9daf23de640d990cfa1415b13e9a44b14"}
Oct 30 11:36:33.192640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.192602    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"]
Oct 30 11:36:33.196883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.196848    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.222309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.222227    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"]
Oct 30 11:36:33.231265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.231235    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 11:36:33.231265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.231233    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-rz9z2\""
Oct 30 11:36:33.231592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.231323    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 11:36:33.257109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.257073    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 11:36:33.314150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.313904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj27k\" (UniqueName: \"kubernetes.io/projected/6530bee4-f8d4-4326-aac2-8cd023b83c91-kube-api-access-jj27k\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.314150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.313966    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6530bee4-f8d4-4326-aac2-8cd023b83c91-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.314150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.314008    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6530bee4-f8d4-4326-aac2-8cd023b83c91-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.314150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.314027    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6530bee4-f8d4-4326-aac2-8cd023b83c91-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.414536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.414505    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj27k\" (UniqueName: \"kubernetes.io/projected/6530bee4-f8d4-4326-aac2-8cd023b83c91-kube-api-access-jj27k\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.414885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.414560    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6530bee4-f8d4-4326-aac2-8cd023b83c91-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.414885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.414607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6530bee4-f8d4-4326-aac2-8cd023b83c91-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.414885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.414632    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6530bee4-f8d4-4326-aac2-8cd023b83c91-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.415594 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.415572    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6530bee4-f8d4-4326-aac2-8cd023b83c91-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.416129 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.416108    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6530bee4-f8d4-4326-aac2-8cd023b83c91-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.416353 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.416333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6530bee4-f8d4-4326-aac2-8cd023b83c91-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.440573 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.440543    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj27k\" (UniqueName: \"kubernetes.io/projected/6530bee4-f8d4-4326-aac2-8cd023b83c91-kube-api-access-jj27k\") pod \"kserve-module-controller-manager-7566cf5fc8-64c77\" (UID: \"6530bee4-f8d4-4326-aac2-8cd023b83c91\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.515391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.515356    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:33.565168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.565136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:33.600524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.600491    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x7t56" event={"ID":"8fda7bf5-994b-47a9-9146-8211c7aeb639","Type":"ContainerStarted","Data":"23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0"}
Oct 30 11:36:33.644347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.644281    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-x7t56" podStartSLOduration=1.013349994 podStartE2EDuration="2.644261904s" totalImagesPullingTime="1.63091191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:33.642167403 +0000 UTC m=+397.437051904" watchObservedRunningTime="2026-10-30 11:36:33.644261904 +0000 UTC m=+397.439146587"
Oct 30 11:36:33.791415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.791337    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:33.836131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.836105    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"]
Oct 30 11:36:33.840629 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:36:33.840600    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6530bee4_f8d4_4326_aac2_8cd023b83c91.slice/crio-1f431846bbba195f9b4fb99b820b16aae1324da38ba5b64dd0b63da85bcff5a2 WatchSource:0}: Error finding container 1f431846bbba195f9b4fb99b820b16aae1324da38ba5b64dd0b63da85bcff5a2: Status 404 returned error can't find the container with id 1f431846bbba195f9b4fb99b820b16aae1324da38ba5b64dd0b63da85bcff5a2
Oct 30 11:36:33.933904 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:33.933861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:34.085752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.085662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:34.370784 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.370691    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:34.527305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.527256    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:34.610732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.610692    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf" event={"ID":"c645ca89-5a90-4a76-9daa-8a1a9818a807","Type":"ContainerStarted","Data":"b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276"}
Oct 30 11:36:34.612368 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.612339    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77" event={"ID":"6530bee4-f8d4-4326-aac2-8cd023b83c91","Type":"ContainerStarted","Data":"1f431846bbba195f9b4fb99b820b16aae1324da38ba5b64dd0b63da85bcff5a2"}
Oct 30 11:36:34.687208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.687101    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf" podStartSLOduration=2.234567164 podStartE2EDuration="3.687081756s" totalImagesPullingTime="1.452514592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:34.685653048 +0000 UTC m=+398.480537543" watchObservedRunningTime="2026-10-30 11:36:34.687081756 +0000 UTC m=+398.481966252"
Oct 30 11:36:34.734788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.734750    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x7t56"]
Oct 30 11:36:34.952204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:34.952164    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:35.617921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:35.617877    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-x7t56" podUID="8fda7bf5-994b-47a9-9146-8211c7aeb639" containerName="authorino" containerID="cri-o://23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0" gracePeriod=30
Oct 30 11:36:35.876723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:35.876639    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:35.942320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:35.942276    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj\" (UniqueName: \"kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj\") pod \"8fda7bf5-994b-47a9-9146-8211c7aeb639\" (UID: \"8fda7bf5-994b-47a9-9146-8211c7aeb639\") "
Oct 30 11:36:35.945236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:35.945195    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj" pod "8fda7bf5-994b-47a9-9146-8211c7aeb639" (UID: "8fda7bf5-994b-47a9-9146-8211c7aeb639"). InnerVolumeSpecName "kube-api-access-d5skj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:36:36.043535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.043499    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5skj\" (UniqueName: \"kubernetes.io/projected/8fda7bf5-994b-47a9-9146-8211c7aeb639-kube-api-access-d5skj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:36:36.623777 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.623742    5060 generic.go:356] "Generic (PLEG): container finished" podID="8fda7bf5-994b-47a9-9146-8211c7aeb639" containerID="23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0" exitCode=0
Oct 30 11:36:36.624283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.623791    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x7t56"
Oct 30 11:36:36.624283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.623833    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x7t56" event={"ID":"8fda7bf5-994b-47a9-9146-8211c7aeb639","Type":"ContainerDied","Data":"23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0"}
Oct 30 11:36:36.624283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.623885    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x7t56" event={"ID":"8fda7bf5-994b-47a9-9146-8211c7aeb639","Type":"ContainerDied","Data":"6c2fdee439e6d03f663e814021ca1ec9daf23de640d990cfa1415b13e9a44b14"}
Oct 30 11:36:36.624283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.623912    5060 scope.go:122] "RemoveContainer" containerID="23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0"
Oct 30 11:36:36.625526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.625493    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77" event={"ID":"6530bee4-f8d4-4326-aac2-8cd023b83c91","Type":"ContainerStarted","Data":"6ec242f33440cfc4898223b3dff2d077d5c1d7cba22d76acff0d19b9f4faf126"}
Oct 30 11:36:36.625627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.625573    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:36:36.632867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.632850    5060 scope.go:122] "RemoveContainer" containerID="23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0"
Oct 30 11:36:36.633138 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:36:36.633121    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0\": container with ID starting with 23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0 not found: ID does not exist" containerID="23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0"
Oct 30 11:36:36.633195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.633147    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0"} err="failed to get container status \"23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0\": rpc error: code = NotFound desc = could not find container \"23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0\": container with ID starting with 23cdf03c012bb0078ec8a0d231f2ea1d21cf738658045622f031d0eda912a7b0 not found: ID does not exist"
Oct 30 11:36:36.683517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.683447    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77" podStartSLOduration=1.411158932 podStartE2EDuration="3.68343437s" totalImagesPullingTime="2.272275438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:36.682108844 +0000 UTC m=+400.476993338" watchObservedRunningTime="2026-10-30 11:36:36.68343437 +0000 UTC m=+400.478318864"
Oct 30 11:36:36.717834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.717795    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x7t56"]
Oct 30 11:36:36.722059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:36.722035    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x7t56"]
Oct 30 11:36:38.332714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:38.332684    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fda7bf5-994b-47a9-9146-8211c7aeb639" path="/var/lib/kubelet/pods/8fda7bf5-994b-47a9-9146-8211c7aeb639/volumes"
Oct 30 11:36:40.588502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:40.588466    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:40.765671 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:40.765639    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:45.358393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:45.358354    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:45.514779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:45.514742    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:46.851463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:46.851419    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:49.856669 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:36:49.856615    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:36:50.847398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:50.847362    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:51.000341 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:51.000305    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:51.741949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:51.741908    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:36:57.632914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:36:57.632881    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-64c77"
Oct 30 11:37:19.857121 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:37:19.857082    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:37:21.360125 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:21.360083    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:21.542752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:21.542715    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:21.732195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:21.732157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:25.405618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:25.405575    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:25.584840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:25.584798    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:28.896818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.896785    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"]
Oct 30 11:37:28.897212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.897164    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8fda7bf5-994b-47a9-9146-8211c7aeb639" containerName="authorino"
Oct 30 11:37:28.897212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.897176    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fda7bf5-994b-47a9-9146-8211c7aeb639" containerName="authorino"
Oct 30 11:37:28.897282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.897255    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fda7bf5-994b-47a9-9146-8211c7aeb639" containerName="authorino"
Oct 30 11:37:28.906044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.906018    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:28.919110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.919077    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"]
Oct 30 11:37:28.922522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.922497    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 11:37:28.922731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.922711    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 11:37:28.928350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.928310    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 11:37:28.928575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:28.928476    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2tbvx\""
Oct 30 11:37:29.013075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.013043    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/436af639-6583-46e3-89d5-665f2c870571-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.013256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.013134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/436af639-6583-46e3-89d5-665f2c870571-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.013256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.013204    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/436af639-6583-46e3-89d5-665f2c870571-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.013256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.013223    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6j7\" (UniqueName: \"kubernetes.io/projected/436af639-6583-46e3-89d5-665f2c870571-kube-api-access-rf6j7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.031325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.031290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:29.113750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.113721    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/436af639-6583-46e3-89d5-665f2c870571-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.113930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.113763    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/436af639-6583-46e3-89d5-665f2c870571-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.113930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.113814    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/436af639-6583-46e3-89d5-665f2c870571-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.113930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.113830    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6j7\" (UniqueName: \"kubernetes.io/projected/436af639-6583-46e3-89d5-665f2c870571-kube-api-access-rf6j7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.114525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.114488    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/436af639-6583-46e3-89d5-665f2c870571-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.114808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.114788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/436af639-6583-46e3-89d5-665f2c870571-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.116404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.116305    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/436af639-6583-46e3-89d5-665f2c870571-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.158241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.158160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6j7\" (UniqueName: \"kubernetes.io/projected/436af639-6583-46e3-89d5-665f2c870571-kube-api-access-rf6j7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wlb56\" (UID: \"436af639-6583-46e3-89d5-665f2c870571\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.222695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.222658    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:29.330940 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.330900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:29.544097 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:37:29.544050    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436af639_6583_46e3_89d5_665f2c870571.slice/crio-c8f42c82637f5f93a2fd77fced0881cc68d4ce83bf6cdd6ae5769f0066d565a7 WatchSource:0}: Error finding container c8f42c82637f5f93a2fd77fced0881cc68d4ce83bf6cdd6ae5769f0066d565a7: Status 404 returned error can't find the container with id c8f42c82637f5f93a2fd77fced0881cc68d4ce83bf6cdd6ae5769f0066d565a7
Oct 30 11:37:29.544215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.544101    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"]
Oct 30 11:37:29.678701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.678670    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:29.868997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.868897    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:29.933654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.933621    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56" event={"ID":"436af639-6583-46e3-89d5-665f2c870571","Type":"ContainerStarted","Data":"c8f42c82637f5f93a2fd77fced0881cc68d4ce83bf6cdd6ae5769f0066d565a7"}
Oct 30 11:37:29.975096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.975060    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kvk9h"]
Oct 30 11:37:29.980201 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:29.980177    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.005321 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.005293    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 11:37:30.008241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.008213    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kvk9h"]
Oct 30 11:37:30.017494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.017444    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-26cxh\""
Oct 30 11:37:30.044578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.044543    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"]
Oct 30 11:37:30.048423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.048399    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.083253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.083223    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"]
Oct 30 11:37:30.087559 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.087528    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:30.107537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.107507    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wnnxb\""
Oct 30 11:37:30.108210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.108186    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 11:37:30.109096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.109077    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-j2qgw"]
Oct 30 11:37:30.113372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.113352    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.124527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.124503    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hckkh\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.124682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.124557    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nn5\" (UniqueName: \"kubernetes.io/projected/c3a2eebf-58c9-42a6-8615-4fde0a678b6f-kube-api-access-r4nn5\") pod \"kserve-controller-manager-776cc66d47-kvk9h\" (UID: \"c3a2eebf-58c9-42a6-8615-4fde0a678b6f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.124682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.124665    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3a2eebf-58c9-42a6-8615-4fde0a678b6f-cert\") pod \"kserve-controller-manager-776cc66d47-kvk9h\" (UID: \"c3a2eebf-58c9-42a6-8615-4fde0a678b6f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.124805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.124693    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsb57\" (UniqueName: \"kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57\") pod \"llmisvc-controller-manager-869d5dcc5d-hckkh\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.132999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.132971    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-j2qgw"]
Oct 30 11:37:30.207929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.207900    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 11:37:30.208160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.208030    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4pzv7\""
Oct 30 11:37:30.225654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.225624    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hckkh\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.225805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.225669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nn5\" (UniqueName: \"kubernetes.io/projected/c3a2eebf-58c9-42a6-8615-4fde0a678b6f-kube-api-access-r4nn5\") pod \"kserve-controller-manager-776cc66d47-kvk9h\" (UID: \"c3a2eebf-58c9-42a6-8615-4fde0a678b6f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.225805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.225706    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3a2eebf-58c9-42a6-8615-4fde0a678b6f-cert\") pod \"kserve-controller-manager-776cc66d47-kvk9h\" (UID: \"c3a2eebf-58c9-42a6-8615-4fde0a678b6f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.225805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.225734    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsb57\" (UniqueName: \"kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57\") pod \"llmisvc-controller-manager-869d5dcc5d-hckkh\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.225805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.225773    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwx9\" (UniqueName: \"kubernetes.io/projected/97e52208-b29d-47e3-b683-f0e835f8e049-kube-api-access-dbwx9\") pod \"model-serving-api-75bd89fdfc-j2qgw\" (UID: \"97e52208-b29d-47e3-b683-f0e835f8e049\") " pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.226030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.225810    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e52208-b29d-47e3-b683-f0e835f8e049-tls-certs\") pod \"model-serving-api-75bd89fdfc-j2qgw\" (UID: \"97e52208-b29d-47e3-b683-f0e835f8e049\") " pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.228335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.228307    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hckkh\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.228487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.228340    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3a2eebf-58c9-42a6-8615-4fde0a678b6f-cert\") pod \"kserve-controller-manager-776cc66d47-kvk9h\" (UID: \"c3a2eebf-58c9-42a6-8615-4fde0a678b6f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.258378 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.258348    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsb57\" (UniqueName: \"kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57\") pod \"llmisvc-controller-manager-869d5dcc5d-hckkh\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.258534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.258421    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nn5\" (UniqueName: \"kubernetes.io/projected/c3a2eebf-58c9-42a6-8615-4fde0a678b6f-kube-api-access-r4nn5\") pod \"kserve-controller-manager-776cc66d47-kvk9h\" (UID: \"c3a2eebf-58c9-42a6-8615-4fde0a678b6f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.268718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.268650    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:30.293575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.293545    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:30.327069 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.327037    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e52208-b29d-47e3-b683-f0e835f8e049-tls-certs\") pod \"model-serving-api-75bd89fdfc-j2qgw\" (UID: \"97e52208-b29d-47e3-b683-f0e835f8e049\") " pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.327252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.327172    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwx9\" (UniqueName: \"kubernetes.io/projected/97e52208-b29d-47e3-b683-f0e835f8e049-kube-api-access-dbwx9\") pod \"model-serving-api-75bd89fdfc-j2qgw\" (UID: \"97e52208-b29d-47e3-b683-f0e835f8e049\") " pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.329804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.329779    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e52208-b29d-47e3-b683-f0e835f8e049-tls-certs\") pod \"model-serving-api-75bd89fdfc-j2qgw\" (UID: \"97e52208-b29d-47e3-b683-f0e835f8e049\") " pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.351615 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.351588    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwx9\" (UniqueName: \"kubernetes.io/projected/97e52208-b29d-47e3-b683-f0e835f8e049-kube-api-access-dbwx9\") pod \"model-serving-api-75bd89fdfc-j2qgw\" (UID: \"97e52208-b29d-47e3-b683-f0e835f8e049\") " pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.360752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.360725    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:30.427065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.426810    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:30.596623 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.596588    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kvk9h"]
Oct 30 11:37:30.597692 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:37:30.597658    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a2eebf_58c9_42a6_8615_4fde0a678b6f.slice/crio-458bd306803feecdf173e266db336ab8bc7ac9e0608643c06e6e2718cd8ff527 WatchSource:0}: Error finding container 458bd306803feecdf173e266db336ab8bc7ac9e0608643c06e6e2718cd8ff527: Status 404 returned error can't find the container with id 458bd306803feecdf173e266db336ab8bc7ac9e0608643c06e6e2718cd8ff527
Oct 30 11:37:30.609682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.609563    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:30.669323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.669293    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"]
Oct 30 11:37:30.733533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.733054    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-j2qgw"]
Oct 30 11:37:30.733847 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:37:30.733809    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e52208_b29d_47e3_b683_f0e835f8e049.slice/crio-a811a5a3fdd2c52206f9fdd96d4060dc835918d7c7e5456cceae65179e58e059 WatchSource:0}: Error finding container a811a5a3fdd2c52206f9fdd96d4060dc835918d7c7e5456cceae65179e58e059: Status 404 returned error can't find the container with id a811a5a3fdd2c52206f9fdd96d4060dc835918d7c7e5456cceae65179e58e059
Oct 30 11:37:30.942603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.942519    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh" event={"ID":"2c8f000b-05b1-4a79-a273-0a9f656a3df7","Type":"ContainerStarted","Data":"2383acae55eafb263dc99d96eac85de328d5f889b78d2745071865ab7b26e0b7"}
Oct 30 11:37:30.944345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.944315    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw" event={"ID":"97e52208-b29d-47e3-b683-f0e835f8e049","Type":"ContainerStarted","Data":"a811a5a3fdd2c52206f9fdd96d4060dc835918d7c7e5456cceae65179e58e059"}
Oct 30 11:37:30.947563 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:30.947535    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h" event={"ID":"c3a2eebf-58c9-42a6-8615-4fde0a678b6f","Type":"ContainerStarted","Data":"458bd306803feecdf173e266db336ab8bc7ac9e0608643c06e6e2718cd8ff527"}
Oct 30 11:37:32.908791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:32.908729    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:33.082865 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:33.082722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:33.234739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:33.234677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:33.382622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:33.382557    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:33.985394 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:33.985359    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw" event={"ID":"97e52208-b29d-47e3-b683-f0e835f8e049","Type":"ContainerStarted","Data":"d0c292bc4c142d30a0f52a99640c465f3b74411fc03bcdfe423c9f13be4e1881"}
Oct 30 11:37:33.985807 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:33.985427    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:34.032170 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:34.030994    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw" podStartSLOduration=1.015668916 podStartE2EDuration="4.03097663s" totalImagesPullingTime="3.015307714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:34.029567666 +0000 UTC m=+457.824452163" watchObservedRunningTime="2026-10-30 11:37:34.03097663 +0000 UTC m=+457.825861123"
Oct 30 11:37:35.005385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:35.004522    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh" event={"ID":"2c8f000b-05b1-4a79-a273-0a9f656a3df7","Type":"ContainerStarted","Data":"1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019"}
Oct 30 11:37:35.005385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:35.005326    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:37:35.007912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:35.007878    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h" event={"ID":"c3a2eebf-58c9-42a6-8615-4fde0a678b6f","Type":"ContainerStarted","Data":"6400fd6a30eb95f4e877185133b924b057fca8dd7cfcf3d32c7c3324c5f7dff4"}
Oct 30 11:37:35.008070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:35.007937    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:37:35.057184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:35.057115    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh" podStartSLOduration=1.5359341359999998 podStartE2EDuration="5.057095154s" totalImagesPullingTime="3.521161018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:35.054958491 +0000 UTC m=+458.849842985" watchObservedRunningTime="2026-10-30 11:37:35.057095154 +0000 UTC m=+458.851979649"
Oct 30 11:37:35.098712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:35.098652    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h" podStartSLOduration=2.248516328 podStartE2EDuration="6.098637365s" totalImagesPullingTime="3.850121037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:35.096525412 +0000 UTC m=+458.891409909" watchObservedRunningTime="2026-10-30 11:37:35.098637365 +0000 UTC m=+458.893521894"
Oct 30 11:37:42.057074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:42.057032    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56" event={"ID":"436af639-6583-46e3-89d5-665f2c870571","Type":"ContainerStarted","Data":"f4be26bb1bb29d2a626c157d83ba36a5e6e90c2943e50568f5eb11c919ab11d7"}
Oct 30 11:37:42.057538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:42.057245    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:42.102468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:42.102410    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56" podStartSLOduration=2.5081347210000002 podStartE2EDuration="14.102397854s" totalImagesPullingTime="11.594263133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:42.10090956 +0000 UTC m=+465.895794055" watchObservedRunningTime="2026-10-30 11:37:42.102397854 +0000 UTC m=+465.897282348"
Oct 30 11:37:45.011909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:45.011878    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-j2qgw"
Oct 30 11:37:46.998838 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:46.998801    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:49.856112 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:37:49.856077    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:37:53.068817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:53.068783    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wlb56"
Oct 30 11:37:53.214864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:53.214826    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:55.877178 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:55.877143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:56.018793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:56.018757    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:57.321359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:57.321324    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:57.503381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:57.503345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:37:57.667131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:37:57.667042    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:38:06.021231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:06.021200    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kvk9h"
Oct 30 11:38:07.024150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:07.024117    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:38:08.115193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:08.115145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:38:08.289926 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:08.289872    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:38:09.186754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.186727    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"]
Oct 30 11:38:09.195578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.195548    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.214177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.214132    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"]
Oct 30 11:38:09.237225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.237185    5060 generic.go:356] "Generic (PLEG): container finished" podID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerID="c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf" exitCode=0
Oct 30 11:38:09.237374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.237250    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" event={"ID":"069c84b3-658d-4e00-a7f1-aebbe5173974","Type":"ContainerDied","Data":"c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"}
Oct 30 11:38:09.237631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.237618    5060 scope.go:122] "RemoveContainer" containerID="c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"
Oct 30 11:38:09.238816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.238786    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:38:09.238913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.238856    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:38:09.238913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.238889    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:38:09.296008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.295976    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296007    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296039    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296113    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5hh\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296159    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296220    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296471    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296501    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.296602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.296586    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.397832    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.397908    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.397935    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.397970    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398513    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398669    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398759 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398729    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398758    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398815    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398846    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5hh\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.398971 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.398898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.399166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.399146    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.399719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.399695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.399879 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.399777    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.403776 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.403717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.404524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.404486    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.423736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.423705    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5hh\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.423883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.423800    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b79r6\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.573512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.573478    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:09.920825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.920794    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"]
Oct 30 11:38:09.921674 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:38:09.921649    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff70f28_f944_41c7_9aa4_ae38ecab3790.slice/crio-f459932596b0cdafbf77940e1b39ec4116c36a1d3a4f0cc11b94a529f8445d80 WatchSource:0}: Error finding container f459932596b0cdafbf77940e1b39ec4116c36a1d3a4f0cc11b94a529f8445d80: Status 404 returned error can't find the container with id f459932596b0cdafbf77940e1b39ec4116c36a1d3a4f0cc11b94a529f8445d80
Oct 30 11:38:09.923855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.923824    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:38:09.923929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.923895    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:38:09.923929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:09.923925    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:38:10.245153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.245117    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" event={"ID":"aff70f28-f944-41c7-9aa4-ae38ecab3790","Type":"ContainerStarted","Data":"082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573"}
Oct 30 11:38:10.245721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.245692    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" event={"ID":"aff70f28-f944-41c7-9aa4-ae38ecab3790","Type":"ContainerStarted","Data":"f459932596b0cdafbf77940e1b39ec4116c36a1d3a4f0cc11b94a529f8445d80"}
Oct 30 11:38:10.247474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.247429    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" event={"ID":"069c84b3-658d-4e00-a7f1-aebbe5173974","Type":"ContainerStarted","Data":"163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4"}
Oct 30 11:38:10.303518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.302327    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podStartSLOduration=1.302293471 podStartE2EDuration="1.302293471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:10.300826428 +0000 UTC m=+494.095710929" watchObservedRunningTime="2026-10-30 11:38:10.302293471 +0000 UTC m=+494.097177967"
Oct 30 11:38:10.573912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.573801    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:10.873279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.873197    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:38:10.873533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.873513    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 11:38:10.873679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:10.873660    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 11:38:11.575722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:11.575690    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b79r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 11:38:11.576120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:11.575757    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 11:38:11.873300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:11.873213    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 11:38:11.873300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:11.873281    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 11:38:12.574698 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:12.574661    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b79r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 11:38:12.574895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:12.574714    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 11:38:12.873354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:12.873248    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 11:38:12.873354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:12.873325    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 11:38:13.574239 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:13.574199    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b79r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 11:38:13.574415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:13.574254    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 11:38:13.735479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:13.734542    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bl7jf"]
Oct 30 11:38:13.737633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:13.737598    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf" podUID="c645ca89-5a90-4a76-9daa-8a1a9818a807" containerName="authorino" containerID="cri-o://b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276" gracePeriod=30
Oct 30 11:38:13.873677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:13.873355    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 11:38:13.873677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:13.873409    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 11:38:14.133689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.133600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:38:14.163174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.163140    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc\" (UniqueName: \"kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc\") pod \"c645ca89-5a90-4a76-9daa-8a1a9818a807\" (UID: \"c645ca89-5a90-4a76-9daa-8a1a9818a807\") "
Oct 30 11:38:14.165891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.165865    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc" pod "c645ca89-5a90-4a76-9daa-8a1a9818a807" (UID: "c645ca89-5a90-4a76-9daa-8a1a9818a807"). InnerVolumeSpecName "kube-api-access-6p4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:38:14.263920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.263892    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4vc\" (UniqueName: \"kubernetes.io/projected/c645ca89-5a90-4a76-9daa-8a1a9818a807-kube-api-access-6p4vc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:38:14.274174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.274142    5060 generic.go:356] "Generic (PLEG): container finished" podID="c645ca89-5a90-4a76-9daa-8a1a9818a807" containerID="b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276" exitCode=0
Oct 30 11:38:14.274322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.274250    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf"
Oct 30 11:38:14.274322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.274283    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf" event={"ID":"c645ca89-5a90-4a76-9daa-8a1a9818a807","Type":"ContainerDied","Data":"b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276"}
Oct 30 11:38:14.274468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.274335    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bl7jf" event={"ID":"c645ca89-5a90-4a76-9daa-8a1a9818a807","Type":"ContainerDied","Data":"5785f261c89ff5e1a87f96f1b6ca862fcc77cd57fc2dbebe1d4ec8830352c2c2"}
Oct 30 11:38:14.274468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.274353    5060 scope.go:122] "RemoveContainer" containerID="b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276"
Oct 30 11:38:14.344970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.344949    5060 scope.go:122] "RemoveContainer" containerID="b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276"
Oct 30 11:38:14.345351 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:38:14.345328    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276\": container with ID starting with b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276 not found: ID does not exist" containerID="b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276"
Oct 30 11:38:14.345480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.345358    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276"} err="failed to get container status \"b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276\": rpc error: code = NotFound desc = could not find container \"b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276\": container with ID starting with b710d7ca618e91cabeb025d62970d3b79b7dc16011e2194390d2adae67074276 not found: ID does not exist"
Oct 30 11:38:14.426950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.426840    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bl7jf"]
Oct 30 11:38:14.429366 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.429335    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bl7jf"]
Oct 30 11:38:14.573951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.573909    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b79r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 11:38:14.574129 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.573967    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 11:38:14.873366 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.873327    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 11:38:14.873574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:14.873376    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 11:38:15.574499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:15.574466    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b79r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 11:38:15.574910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:15.574517    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 11:38:15.881558 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:15.876547    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:38:15.881558 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:15.877749    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:38:15.894475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:15.888784    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:38:16.334652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:16.334621    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c645ca89-5a90-4a76-9daa-8a1a9818a807" path="/var/lib/kubelet/pods/c645ca89-5a90-4a76-9daa-8a1a9818a807/volumes"
Oct 30 11:38:16.578195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:16.578166    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:16.578700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:16.578446    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:16.579187 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:16.579167    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:38:19.856330 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:38:19.856294    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:38:40.055505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.055468    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"]
Oct 30 11:38:40.056947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.056920    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c645ca89-5a90-4a76-9daa-8a1a9818a807" containerName="authorino"
Oct 30 11:38:40.057131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.057085    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c645ca89-5a90-4a76-9daa-8a1a9818a807" containerName="authorino"
Oct 30 11:38:40.057517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.057499    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c645ca89-5a90-4a76-9daa-8a1a9818a807" containerName="authorino"
Oct 30 11:38:40.061175 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.061156    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.076706 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.076675    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"]
Oct 30 11:38:40.211716 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.211679    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08df536b-fbf0-44f9-ae9f-ef64ee725386-cert\") pod \"llmisvc-controller-manager-6466c554d8-8t6zk\" (UID: \"08df536b-fbf0-44f9-ae9f-ef64ee725386\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.211895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.211745    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxj8z\" (UniqueName: \"kubernetes.io/projected/08df536b-fbf0-44f9-ae9f-ef64ee725386-kube-api-access-hxj8z\") pod \"llmisvc-controller-manager-6466c554d8-8t6zk\" (UID: \"08df536b-fbf0-44f9-ae9f-ef64ee725386\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.312900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.312818    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxj8z\" (UniqueName: \"kubernetes.io/projected/08df536b-fbf0-44f9-ae9f-ef64ee725386-kube-api-access-hxj8z\") pod \"llmisvc-controller-manager-6466c554d8-8t6zk\" (UID: \"08df536b-fbf0-44f9-ae9f-ef64ee725386\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.312900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.312889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08df536b-fbf0-44f9-ae9f-ef64ee725386-cert\") pod \"llmisvc-controller-manager-6466c554d8-8t6zk\" (UID: \"08df536b-fbf0-44f9-ae9f-ef64ee725386\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.315388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.315360    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08df536b-fbf0-44f9-ae9f-ef64ee725386-cert\") pod \"llmisvc-controller-manager-6466c554d8-8t6zk\" (UID: \"08df536b-fbf0-44f9-ae9f-ef64ee725386\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.336812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.336785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxj8z\" (UniqueName: \"kubernetes.io/projected/08df536b-fbf0-44f9-ae9f-ef64ee725386-kube-api-access-hxj8z\") pod \"llmisvc-controller-manager-6466c554d8-8t6zk\" (UID: \"08df536b-fbf0-44f9-ae9f-ef64ee725386\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.374153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.374122    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:40.673196 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:38:40.673157    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08df536b_fbf0_44f9_ae9f_ef64ee725386.slice/crio-1046e749b6270fb5249a9a16517d29479376f486476606a68ee748e57d243320 WatchSource:0}: Error finding container 1046e749b6270fb5249a9a16517d29479376f486476606a68ee748e57d243320: Status 404 returned error can't find the container with id 1046e749b6270fb5249a9a16517d29479376f486476606a68ee748e57d243320
Oct 30 11:38:40.674269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:40.674243    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"]
Oct 30 11:38:41.446396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:41.446362    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk" event={"ID":"08df536b-fbf0-44f9-ae9f-ef64ee725386","Type":"ContainerStarted","Data":"bc5e94467c3c82f283330bf28a10ecdcd58a24588b508598a793d2047e3215de"}
Oct 30 11:38:41.446837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:41.446417    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk" event={"ID":"08df536b-fbf0-44f9-ae9f-ef64ee725386","Type":"ContainerStarted","Data":"1046e749b6270fb5249a9a16517d29479376f486476606a68ee748e57d243320"}
Oct 30 11:38:41.446837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:41.446484    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:38:41.487617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:41.487562    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk" podStartSLOduration=1.175722152 podStartE2EDuration="1.487547964s" totalImagesPullingTime="311.825812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:41.486802151 +0000 UTC m=+525.281686645" watchObservedRunningTime="2026-10-30 11:38:41.487547964 +0000 UTC m=+525.282432488"
Oct 30 11:38:47.123598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:47.123555    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:38:48.531431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:48.531393    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:38:48.674168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:38:48.674129    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:38:49.855938 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:38:49.855902    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:39:12.454803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.454772    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8t6zk"
Oct 30 11:39:12.531970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.531935    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"]
Oct 30 11:39:12.532256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.532207    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh" podUID="2c8f000b-05b1-4a79-a273-0a9f656a3df7" containerName="manager" containerID="cri-o://1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019" gracePeriod=30
Oct 30 11:39:12.786033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.785995    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:39:12.891611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.891579    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57\" (UniqueName: \"kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57\") pod \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") "
Oct 30 11:39:12.891810 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.891623    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert\" (UniqueName: \"kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert\") pod \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\" (UID: \"2c8f000b-05b1-4a79-a273-0a9f656a3df7\") "
Oct 30 11:39:12.893973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.893945    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert" pod "2c8f000b-05b1-4a79-a273-0a9f656a3df7" (UID: "2c8f000b-05b1-4a79-a273-0a9f656a3df7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:39:12.894128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.893983    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57" pod "2c8f000b-05b1-4a79-a273-0a9f656a3df7" (UID: "2c8f000b-05b1-4a79-a273-0a9f656a3df7"). InnerVolumeSpecName "kube-api-access-fsb57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:39:12.992351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.992269    5060 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8f000b-05b1-4a79-a273-0a9f656a3df7-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:39:12.992351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:12.992299    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsb57\" (UniqueName: \"kubernetes.io/projected/2c8f000b-05b1-4a79-a273-0a9f656a3df7-kube-api-access-fsb57\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:39:13.617295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.617258    5060 generic.go:356] "Generic (PLEG): container finished" podID="2c8f000b-05b1-4a79-a273-0a9f656a3df7" containerID="1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019" exitCode=0
Oct 30 11:39:13.617735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.617298    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"
Oct 30 11:39:13.617735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.617336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh" event={"ID":"2c8f000b-05b1-4a79-a273-0a9f656a3df7","Type":"ContainerDied","Data":"1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019"}
Oct 30 11:39:13.617735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.617390    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh" event={"ID":"2c8f000b-05b1-4a79-a273-0a9f656a3df7","Type":"ContainerDied","Data":"2383acae55eafb263dc99d96eac85de328d5f889b78d2745071865ab7b26e0b7"}
Oct 30 11:39:13.617735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.617415    5060 scope.go:122] "RemoveContainer" containerID="1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019"
Oct 30 11:39:13.626978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.626961    5060 scope.go:122] "RemoveContainer" containerID="1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019"
Oct 30 11:39:13.627244 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:39:13.627227    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019\": container with ID starting with 1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019 not found: ID does not exist" containerID="1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019"
Oct 30 11:39:13.627310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.627252    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019"} err="failed to get container status \"1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019\": rpc error: code = NotFound desc = could not find container \"1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019\": container with ID starting with 1de69064657d80f893c620bd1c8a1856a10a3402b5b4ccd3caa28f95164e8019 not found: ID does not exist"
Oct 30 11:39:13.673281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.673248    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"]
Oct 30 11:39:13.685758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.685727    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hckkh"]
Oct 30 11:39:13.687412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.687370    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:13.851781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:13.851732    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:14.333478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:14.333426    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8f000b-05b1-4a79-a273-0a9f656a3df7" path="/var/lib/kubelet/pods/2c8f000b-05b1-4a79-a273-0a9f656a3df7/volumes"
Oct 30 11:39:17.216021 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:17.215990    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:17.392167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:17.392134    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:19.856846 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:39:19.856810    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:39:42.793115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:42.793017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:43.318794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:43.318757    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:44.605895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:44.605843    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:44.765564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:44.765512    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:47.247280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:47.247240    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:48.887303 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:48.887266    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:49.864063 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:39:49.864026    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:39:50.487788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:50.487754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:52.000854 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:52.000815    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:53.170601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:53.170564    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:55.154978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:55.154929    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:56.396413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:56.396369    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:39:56.398863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:56.398841    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:39:57.058487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:57.058419    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:39:59.151331 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:39:59.151298    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:00.538000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:00.537963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:02.593915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:02.593875    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:19.856199 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:40:19.856156    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:40:20.293510 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:20.293437    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:20.484836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:20.484800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:21.869418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:21.869380    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:22.027624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:22.027591    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:24.539674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.539644    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"]
Oct 30 11:40:24.540073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.540034    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2c8f000b-05b1-4a79-a273-0a9f656a3df7" containerName="manager"
Oct 30 11:40:24.540073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.540045    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8f000b-05b1-4a79-a273-0a9f656a3df7" containerName="manager"
Oct 30 11:40:24.540149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.540126    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8f000b-05b1-4a79-a273-0a9f656a3df7" containerName="manager"
Oct 30 11:40:24.543294 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.543275    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.576545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.576515    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"]
Oct 30 11:40:24.588644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588617    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588670    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588703    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588749    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588788    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588823    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588843    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588875    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h7d\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.588910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.588904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689369    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689405    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689442    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689513    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689556    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689605    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689629    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689667    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h7d\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.689965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.689840    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.690105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.690080    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.690238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.690211    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.690238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.690234    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.690407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.690321    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.691991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.691974    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.692368 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.692346    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.713750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.713721    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.713878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.713750    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h7d\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnsct\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:24.857019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:24.856942    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:25.177339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:25.177315    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"]
Oct 30 11:40:25.178340 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:40:25.178314    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1232d7f_1026_4ff6_942f_aa8c210ccd42.slice/crio-ae054a9f9973fd089985f219cbe055a1aa5fe5c628db2189199be34460adf4d2 WatchSource:0}: Error finding container ae054a9f9973fd089985f219cbe055a1aa5fe5c628db2189199be34460adf4d2: Status 404 returned error can't find the container with id ae054a9f9973fd089985f219cbe055a1aa5fe5c628db2189199be34460adf4d2
Oct 30 11:40:25.180480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:25.180425    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:40:25.180545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:25.180516    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:40:25.180582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:25.180550    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:40:26.047229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:26.047190    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" event={"ID":"a1232d7f-1026-4ff6-942f-aa8c210ccd42","Type":"ContainerStarted","Data":"fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb"}
Oct 30 11:40:26.047773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:26.047249    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" event={"ID":"a1232d7f-1026-4ff6-942f-aa8c210ccd42","Type":"ContainerStarted","Data":"ae054a9f9973fd089985f219cbe055a1aa5fe5c628db2189199be34460adf4d2"}
Oct 30 11:40:26.113032 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:26.112982    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podStartSLOduration=2.112967381 podStartE2EDuration="2.112967381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:26.111206731 +0000 UTC m=+629.906091226" watchObservedRunningTime="2026-10-30 11:40:26.112967381 +0000 UTC m=+629.907851949"
Oct 30 11:40:26.857559 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:26.857520    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:26.859161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:26.859130    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnsct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 11:40:26.859298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:26.859181    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 11:40:27.858337 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:27.858294    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnsct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 11:40:27.858878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:27.858358    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 11:40:28.858329 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:28.858292    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnsct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 11:40:28.858554 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:28.858359    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 11:40:29.857551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:29.857512    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnsct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 11:40:29.857739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:29.857577    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 11:40:30.857315 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:30.857270    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnsct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 11:40:30.857751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:30.857331    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 11:40:31.858366 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:31.858318    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnsct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 11:40:31.858944 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:31.858387    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 11:40:33.416263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:33.416231    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:33.416816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:33.416647    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:33.417336 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:33.417311    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:40:41.107081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:41.107044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:41.249258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:41.249224    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:47.382716 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:40:47.382659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:40:49.855198 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:40:49.855163    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:41:19.856650 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:41:19.856603    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:41:38.193708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:38.193651    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:38.360719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:38.360681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:38.524274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:38.524240    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:38.750283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:38.750246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:38.990446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:38.990406    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:39.261466 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:39.261357    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:39.416404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:39.416372    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:39.560800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:39.560712    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:40.195055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:40.195023    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:40.683718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:40.683674    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:40.832484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:40.832428    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:41.305540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:41.305508    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:41.445933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:41.445889    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:41.973941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:41.973900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:43.060242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:43.060209    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:43.252106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:43.252054    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:43.554625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:43.554586    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:43.716189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:43.716148    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:44.477856 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:44.477817    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:44.685984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:44.685951    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:44.978465 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:44.978408    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:45.142791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:45.142758    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:45.546043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:45.546011    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:45.687668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:45.687630    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:47.441093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:47.441045    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:47.610591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:47.610551    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:47.754334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:47.754296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:48.106948 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.106849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:48.618498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.618432    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:48.762807 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.762772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:48.886390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.886307    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"]
Oct 30 11:41:48.890188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.890165    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.918626 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.918597    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"]
Oct 30 11:41:48.957404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957368    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957419    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957607    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957740    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957770    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vg5\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957833    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957899    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.957975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.957945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:48.990930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.990884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:48.993224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:48.993205    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-90364398-openshift-default-dockercfg-tjkms\""
Oct 30 11:41:49.058899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.058865    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.058899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.058903    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.058935    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.058962    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.058981    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.058997    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vg5\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.059029    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.059045    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.059064    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.059574    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.059986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.059964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.060122 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.060099    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.060256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.060236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.060626 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.060605    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.062711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.062684    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.063244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.063217    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.083377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.083355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vg5\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.083377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.083364    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token\") pod \"e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") " pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.206047 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.205999    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:49.250870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.250831    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:49.437883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.437847    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:49.533115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.533088    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"]
Oct 30 11:41:49.533283 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:41:49.533265    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0510329a_49ca_408c_a5ad_1a508dfd86c4.slice/crio-fe557b32740ba342ba5f185ecff53e12740b6e9d69b3bfb51f13842172efb4fd WatchSource:0}: Error finding container fe557b32740ba342ba5f185ecff53e12740b6e9d69b3bfb51f13842172efb4fd: Status 404 returned error can't find the container with id fe557b32740ba342ba5f185ecff53e12740b6e9d69b3bfb51f13842172efb4fd
Oct 30 11:41:49.535493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.535432    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:41:49.535567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.535533    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:41:49.535605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.535568    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:41:49.587157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.587115    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:49.771391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:49.771353    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:49.860389 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:41:49.860303    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:41:50.518781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:50.518747    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" event={"ID":"0510329a-49ca-408c-a5ad-1a508dfd86c4","Type":"ContainerStarted","Data":"80149a64372ddcc454b29f9624fd16f91125372a67676e17cf8ca958107f416c"}
Oct 30 11:41:50.518991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:50.518803    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" event={"ID":"0510329a-49ca-408c-a5ad-1a508dfd86c4","Type":"ContainerStarted","Data":"fe557b32740ba342ba5f185ecff53e12740b6e9d69b3bfb51f13842172efb4fd"}
Oct 30 11:41:50.580544 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:50.580490    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podStartSLOduration=2.580477029 podStartE2EDuration="2.580477029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:50.578117457 +0000 UTC m=+714.373001946" watchObservedRunningTime="2026-10-30 11:41:50.580477029 +0000 UTC m=+714.375361522"
Oct 30 11:41:51.076189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.076157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:51.211382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.211299    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:51.226911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.226883    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:51.230536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.230490    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:51.529770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.529735    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:51.530724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.530699    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:41:51.702876 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:51.702848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:52.731344 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:52.731304    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:52.890990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:52.890957    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:53.041301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.041207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:53.075199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.075165    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"]
Oct 30 11:41:53.078999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.078981    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.097406 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.097377    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"]
Oct 30 11:41:53.106593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.106560    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-64715175\""
Oct 30 11:41:53.106759 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.106627    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-64715175-dockercfg-n76wd\""
Oct 30 11:41:53.202197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.202161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfsz\" (UniqueName: \"kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz\") pod \"payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.202399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.202308    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.303446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.303341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.303446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.303435    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfsz\" (UniqueName: \"kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz\") pod \"payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.304254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.304227    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.328245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.328210    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfsz\" (UniqueName: \"kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz\") pod \"payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.400474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.394335    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:53.704984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.704953    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"]
Oct 30 11:41:53.705520 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:41:53.705491    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a5c283_52e6_4d0b_8941_22c9b95144f2.slice/crio-c1c16b7770f44752a08ffb2478d7c54fa47700d00b3b00e68883fcd81dd88449 WatchSource:0}: Error finding container c1c16b7770f44752a08ffb2478d7c54fa47700d00b3b00e68883fcd81dd88449: Status 404 returned error can't find the container with id c1c16b7770f44752a08ffb2478d7c54fa47700d00b3b00e68883fcd81dd88449
Oct 30 11:41:53.707636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.707617    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:41:53.737615 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:53.737563    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:54.549638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:54.549597    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q" event={"ID":"c1a5c283-52e6-4d0b-8941-22c9b95144f2","Type":"ContainerStarted","Data":"c1c16b7770f44752a08ffb2478d7c54fa47700d00b3b00e68883fcd81dd88449"}
Oct 30 11:41:55.686147 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:55.686112    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:55.858484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:55.858437    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:56.564662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:56.564606    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q" event={"ID":"c1a5c283-52e6-4d0b-8941-22c9b95144f2","Type":"ContainerStarted","Data":"6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823"}
Oct 30 11:41:56.564855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:56.564812    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:41:56.606057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:56.606001    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q" podStartSLOduration=1.55597132 podStartE2EDuration="3.605989043s" totalImagesPullingTime="2.050017723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:56.605433024 +0000 UTC m=+720.400317517" watchObservedRunningTime="2026-10-30 11:41:56.605989043 +0000 UTC m=+720.400873536"
Oct 30 11:41:57.454630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.454599    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"]
Oct 30 11:41:57.458561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.458538    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.476264 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.476238    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"]
Oct 30 11:41:57.485589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.485564    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:41:57.485589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.485562    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:41:57.485804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.485736    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:41:57.485804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.485756    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:41:57.548663 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.548624    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.548851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.548672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.548851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.548772    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49n4\" (UniqueName: \"kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.650016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.649966    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.650016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.650015    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.650279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.650043    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49n4\" (UniqueName: \"kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.651082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.651041    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.653259 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.653231    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.674090 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.674058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49n4\" (UniqueName: \"kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4\") pod \"maas-api-revoke-keys-e2e-ait-0836c883-kngq8\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:57.757302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.757199    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:41:57.771171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:57.771138    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:41:58.077432 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:41:58.077398    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387331cc_eac4_4567_afa1_c2979837034a.slice/crio-1069d065b8591b8baa7d4f4d4e776f223913434fd74f17a8f06e9b09097f7daa WatchSource:0}: Error finding container 1069d065b8591b8baa7d4f4d4e776f223913434fd74f17a8f06e9b09097f7daa: Status 404 returned error can't find the container with id 1069d065b8591b8baa7d4f4d4e776f223913434fd74f17a8f06e9b09097f7daa
Oct 30 11:41:58.077576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:58.077438    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"]
Oct 30 11:41:58.578072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:58.578036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8" event={"ID":"387331cc-eac4-4567-afa1-c2979837034a","Type":"ContainerStarted","Data":"1069d065b8591b8baa7d4f4d4e776f223913434fd74f17a8f06e9b09097f7daa"}
Oct 30 11:41:59.973793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:41:59.973754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:00.116699 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:00.116604    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:00.262215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:00.262188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:01.600164 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:01.600128    5060 generic.go:356] "Generic (PLEG): container finished" podID="387331cc-eac4-4567-afa1-c2979837034a" containerID="50d7e0b3730de05d32b10b66fd07a08bd4f2531924ee2832eafa8fa77eebce72" exitCode=0
Oct 30 11:42:01.600594 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:01.600217    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8" event={"ID":"387331cc-eac4-4567-afa1-c2979837034a","Type":"ContainerDied","Data":"50d7e0b3730de05d32b10b66fd07a08bd4f2531924ee2832eafa8fa77eebce72"}
Oct 30 11:42:01.927772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:01.927687    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:02.074470 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.074423    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:02.213229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.213197    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:02.764621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.764596    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:42:02.909399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.909318    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca\") pod \"387331cc-eac4-4567-afa1-c2979837034a\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") "
Oct 30 11:42:02.909399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.909355    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4\" (UniqueName: \"kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4\") pod \"387331cc-eac4-4567-afa1-c2979837034a\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") "
Oct 30 11:42:02.909619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.909442    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca\") pod \"387331cc-eac4-4567-afa1-c2979837034a\" (UID: \"387331cc-eac4-4567-afa1-c2979837034a\") "
Oct 30 11:42:02.909752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.909728    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca" pod "387331cc-eac4-4567-afa1-c2979837034a" (UID: "387331cc-eac4-4567-afa1-c2979837034a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:02.911808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.911783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca" pod "387331cc-eac4-4567-afa1-c2979837034a" (UID: "387331cc-eac4-4567-afa1-c2979837034a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:02.911915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:02.911865    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4" pod "387331cc-eac4-4567-afa1-c2979837034a" (UID: "387331cc-eac4-4567-afa1-c2979837034a"). InnerVolumeSpecName "kube-api-access-s49n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:03.014479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.010671    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387331cc-eac4-4567-afa1-c2979837034a-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:03.014479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.010709    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387331cc-eac4-4567-afa1-c2979837034a-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:03.014479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.010724    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s49n4\" (UniqueName: \"kubernetes.io/projected/387331cc-eac4-4567-afa1-c2979837034a-kube-api-access-s49n4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:03.614357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.614327    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"
Oct 30 11:42:03.614357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.614353    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8" event={"ID":"387331cc-eac4-4567-afa1-c2979837034a","Type":"ContainerDied","Data":"1069d065b8591b8baa7d4f4d4e776f223913434fd74f17a8f06e9b09097f7daa"}
Oct 30 11:42:03.614584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.614395    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1069d065b8591b8baa7d4f4d4e776f223913434fd74f17a8f06e9b09097f7daa"
Oct 30 11:42:03.867710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:03.867617    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:04.947088 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.947053    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"]
Oct 30 11:42:04.947562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.947547    5060 cpu_manager.go:414] "containerMap: removing container" podUID="387331cc-eac4-4567-afa1-c2979837034a" containerName="revoke-keys"
Oct 30 11:42:04.947616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.947565    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387331cc-eac4-4567-afa1-c2979837034a" containerName="revoke-keys"
Oct 30 11:42:04.947682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.947672    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="387331cc-eac4-4567-afa1-c2979837034a" containerName="revoke-keys"
Oct 30 11:42:04.951140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.951113    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:04.970568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.970542    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"]
Oct 30 11:42:04.980161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.980129    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:42:04.980161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.980150    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:42:04.980353 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.980150    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:42:04.980353 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:04.980216    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:42:05.130834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.130801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.131049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.130857    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.131049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.130956    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7sv\" (UniqueName: \"kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.232142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.232056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.232142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.232107    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7sv\" (UniqueName: \"kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.232342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.232177    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.232891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.232837    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.234700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.234668    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.261603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.261578    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7sv\" (UniqueName: \"kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv\") pod \"maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.265463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.265433    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:05.571020 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.570992    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"]
Oct 30 11:42:05.571283 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:42:05.571263    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71cdb67_8bf4_4bfa_83ef_2e55f62c13f2.slice/crio-61413e6e2d7a9b3a6c7f017f7bb3fef1752eb27fb4137ad8c9d842ecb0511ee6 WatchSource:0}: Error finding container 61413e6e2d7a9b3a6c7f017f7bb3fef1752eb27fb4137ad8c9d842ecb0511ee6: Status 404 returned error can't find the container with id 61413e6e2d7a9b3a6c7f017f7bb3fef1752eb27fb4137ad8c9d842ecb0511ee6
Oct 30 11:42:05.627468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:05.627421    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg" event={"ID":"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2","Type":"ContainerStarted","Data":"61413e6e2d7a9b3a6c7f017f7bb3fef1752eb27fb4137ad8c9d842ecb0511ee6"}
Oct 30 11:42:06.070708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:06.070662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:06.634603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:06.634571    5060 generic.go:356] "Generic (PLEG): container finished" podID="f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" containerID="e2c6d5256cd4efaae6b7545e0bbaa82fa01781fe9d6aa1c00013aaea64df2aa2" exitCode=0
Oct 30 11:42:06.634772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:06.634655    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg" event={"ID":"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2","Type":"ContainerDied","Data":"e2c6d5256cd4efaae6b7545e0bbaa82fa01781fe9d6aa1c00013aaea64df2aa2"}
Oct 30 11:42:07.573934 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.573903    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:42:07.786521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.786493    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:07.854863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.854774    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca\") pod \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") "
Oct 30 11:42:07.855028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.854861    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv\" (UniqueName: \"kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv\") pod \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") "
Oct 30 11:42:07.855028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.854927    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca\") pod \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\" (UID: \"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2\") "
Oct 30 11:42:07.855294 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.855262    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca" pod "f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" (UID: "f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:07.857217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.857188    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca" pod "f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" (UID: "f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:07.857298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.857243    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv" pod "f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" (UID: "f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2"). InnerVolumeSpecName "kube-api-access-wg7sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:07.956305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.956269    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:07.956305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.956300    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg7sv\" (UniqueName: \"kubernetes.io/projected/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-kube-api-access-wg7sv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:07.956305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:07.956310    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:08.184376 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:08.184279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:08.331982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:08.331948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:08.660477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:08.660419    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg" event={"ID":"f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2","Type":"ContainerDied","Data":"61413e6e2d7a9b3a6c7f017f7bb3fef1752eb27fb4137ad8c9d842ecb0511ee6"}
Oct 30 11:42:08.660477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:08.660478    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"
Oct 30 11:42:08.660984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:08.660500    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61413e6e2d7a9b3a6c7f017f7bb3fef1752eb27fb4137ad8c9d842ecb0511ee6"
Oct 30 11:42:09.084084 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:09.084047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:09.226416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:09.226382    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:10.423531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:10.423500    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:12.268552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:12.268513    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:14.063723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:14.063689    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:15.342788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:15.342749    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:15.498415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:15.498384    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:15.783408 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:15.783320    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:16.303272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:16.303234    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:16.458475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:16.458422    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:16.538702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:16.538663    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"]
Oct 30 11:42:16.550723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:16.550688    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0836c883-kngq8"]
Oct 30 11:42:16.607442 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:16.607316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:16.920789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:16.920673    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:17.074231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:17.074195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:17.234108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:17.234076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:17.373292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:17.373244    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:18.107815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.107777    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:18.333433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.333403    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387331cc-eac4-4567-afa1-c2979837034a" path="/var/lib/kubelet/pods/387331cc-eac4-4567-afa1-c2979837034a/volumes"
Oct 30 11:42:18.587425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.587391    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:18.751179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.751143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:18.757014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.756989    5060 generic.go:356] "Generic (PLEG): container finished" podID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerID="80149a64372ddcc454b29f9624fd16f91125372a67676e17cf8ca958107f416c" exitCode=0
Oct 30 11:42:18.757194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.757070    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" event={"ID":"0510329a-49ca-408c-a5ad-1a508dfd86c4","Type":"ContainerDied","Data":"80149a64372ddcc454b29f9624fd16f91125372a67676e17cf8ca958107f416c"}
Oct 30 11:42:18.757538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.757519    5060 scope.go:122] "RemoveContainer" containerID="80149a64372ddcc454b29f9624fd16f91125372a67676e17cf8ca958107f416c"
Oct 30 11:42:18.760120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.760088    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:18.760213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.760177    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:18.760262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:18.760221    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:19.102060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.102021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:19.669867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.669840    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:19.765380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.765336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" event={"ID":"0510329a-49ca-408c-a5ad-1a508dfd86c4","Type":"ContainerStarted","Data":"f5dd357385e1ebbf443f9c2e15edb8e8d9ae205d9af68aed5c04c8ce9c4feacc"}
Oct 30 11:42:19.810004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.809962    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"]
Oct 30 11:42:19.818146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.818108    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a5c09a-gtvxg"]
Oct 30 11:42:19.828794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.828742    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:19.862600 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:42:19.862564    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:42:19.967682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:19.967645    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:20.207765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:20.207642    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:42:20.208119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:20.208097    5060 patch_prober.go:29] interesting pod/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 11:42:20.208321 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:20.208282    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 11:42:20.334044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:20.333957    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" path="/var/lib/kubelet/pods/f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2/volumes"
Oct 30 11:42:21.207963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:21.207920    5060 patch_prober.go:29] interesting pod/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 11:42:21.208552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:21.207991    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 11:42:21.530598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:21.530513    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:42:22.108147 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:22.108107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:22.208941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:22.207117    5060 patch_prober.go:29] interesting pod/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 11:42:22.208941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:22.207191    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 11:42:22.268588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:22.268556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:23.207377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:23.207343    5060 patch_prober.go:29] interesting pod/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 11:42:23.207627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:23.207392    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 11:42:23.819297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:23.819262    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:24.002077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:24.002043    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:24.208596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:24.208556    5060 patch_prober.go:29] interesting pod/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 11:42:24.208830 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:24.208639    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 11:42:24.209648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:24.209604    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:25.213288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:25.213259    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:42:25.214554 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:25.214530    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:42:26.049784 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:26.049741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:27.961503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:27.961467    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:28.142915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:28.142881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:28.297284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:28.297182    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:28.560170 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:28.560074    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:28.710204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:28.710168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:29.040313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:29.040281    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:29.540837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:29.540796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:29.741218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:29.741185    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:29.893973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:29.893881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:30.432018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:30.431977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:30.578719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:30.578678    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:30.751439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:30.751407    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"]
Oct 30 11:42:30.751825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:30.751673    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy" containerID="cri-o://f5dd357385e1ebbf443f9c2e15edb8e8d9ae205d9af68aed5c04c8ce9c4feacc" gracePeriod=30
Oct 30 11:42:30.758976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:30.758946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:30.920812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:30.920776    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:31.334895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:31.334854    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:31.649918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:31.649819    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:31.844967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:31.844934    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:32.076481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.069559    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:32.136822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.136793    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"]
Oct 30 11:42:32.137273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.137259    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" containerName="revoke-keys"
Oct 30 11:42:32.137358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.137274    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" containerName="revoke-keys"
Oct 30 11:42:32.137358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.137348    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71cdb67-8bf4-4bfa-83ef-2e55f62c13f2" containerName="revoke-keys"
Oct 30 11:42:32.146645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.146618    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.167882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.167854    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"]
Oct 30 11:42:32.176211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.176182    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1d9b1a-openshift-default-dockercfg-p2zdz\""
Oct 30 11:42:32.247404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247361    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247521    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247550    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6cf\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247657    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247785    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247867    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.247921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.247894    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.256825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.256793    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:32.348440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348364    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348416    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348437    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6cf\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348471    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348649    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348697    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348775    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348866    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.348973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.348899    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.349085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.349049    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.349235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.349213    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.349299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.349232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.349337 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.349306    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.349417 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.349397    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.351095 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.351074    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.351189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.351075    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.372794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.372763    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.373036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.373013    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6cf\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.451770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.451735    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:32.469616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.469584    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:32.779045 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.778997    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:32.798650 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.798624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"]
Oct 30 11:42:32.798824 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:42:32.798801    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2248c34_0ee8_4d0a_8b6d_0233e3a29790.slice/crio-34ce250e934efce5910ac99a7815bb78feff437646fc70ab33f89e376783a1e5 WatchSource:0}: Error finding container 34ce250e934efce5910ac99a7815bb78feff437646fc70ab33f89e376783a1e5: Status 404 returned error can't find the container with id 34ce250e934efce5910ac99a7815bb78feff437646fc70ab33f89e376783a1e5
Oct 30 11:42:32.801864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.801832    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:32.801952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.801904    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:32.801952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.801932    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:32.885638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.885595    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" event={"ID":"d2248c34-0ee8-4d0a-8b6d-0233e3a29790","Type":"ContainerStarted","Data":"9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"}
Oct 30 11:42:32.885896 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.885665    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" event={"ID":"d2248c34-0ee8-4d0a-8b6d-0233e3a29790","Type":"ContainerStarted","Data":"34ce250e934efce5910ac99a7815bb78feff437646fc70ab33f89e376783a1e5"}
Oct 30 11:42:32.937820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:32.937756    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podStartSLOduration=0.937737607 podStartE2EDuration="937.737607ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:32.934338145 +0000 UTC m=+756.729222639" watchObservedRunningTime="2026-10-30 11:42:32.937737607 +0000 UTC m=+756.732622105"
Oct 30 11:42:33.264263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:33.264227    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:33.428424 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:33.428389    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:33.469874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:33.469840    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:33.474641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:33.474619    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:33.890840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:33.890811    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:33.891841 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:33.891822    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:34.992166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:34.992133    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:35.148400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.148357    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:35.309933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.309465    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:35.533014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.532981    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:35.705528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.705494    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:35.914019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.913988    5060 generic.go:356] "Generic (PLEG): container finished" podID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerID="f5dd357385e1ebbf443f9c2e15edb8e8d9ae205d9af68aed5c04c8ce9c4feacc" exitCode=0
Oct 30 11:42:35.914243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.914084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" event={"ID":"0510329a-49ca-408c-a5ad-1a508dfd86c4","Type":"ContainerDied","Data":"f5dd357385e1ebbf443f9c2e15edb8e8d9ae205d9af68aed5c04c8ce9c4feacc"}
Oct 30 11:42:35.914243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:35.914163    5060 scope.go:122] "RemoveContainer" containerID="80149a64372ddcc454b29f9624fd16f91125372a67676e17cf8ca958107f416c"
Oct 30 11:42:36.034774 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.034742    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:42:36.069928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.069894    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:36.087842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.087808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.087858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.087892    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.087928    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.087988    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088022    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088082    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088112    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:36.088295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088164    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:36.088295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088173    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088244    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo\") pod \"0510329a-49ca-408c-a5ad-1a508dfd86c4\" (UID: \"0510329a-49ca-408c-a5ad-1a508dfd86c4\") "
Oct 30 11:42:36.088658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088382    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:36.088658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088524    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:36.088658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:36.088802 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088783    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.088860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088803    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.088860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088818    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0510329a-49ca-408c-a5ad-1a508dfd86c4-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.088860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088834    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.088860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.088847    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.090775 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.090750    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:42:36.090899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.090872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:36.090951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.090898    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "kube-api-access-h8vg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:36.090951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.090918    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token" pod "0510329a-49ca-408c-a5ad-1a508dfd86c4" (UID: "0510329a-49ca-408c-a5ad-1a508dfd86c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:36.190110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.190066    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.190110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.190102    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8vg5\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-kube-api-access-h8vg5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.190110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.190115    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.190354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.190127    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0510329a-49ca-408c-a5ad-1a508dfd86c4-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:36.220733 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.220700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:36.921355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.921325    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"
Oct 30 11:42:36.921355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.921340    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5" event={"ID":"0510329a-49ca-408c-a5ad-1a508dfd86c4","Type":"ContainerDied","Data":"fe557b32740ba342ba5f185ecff53e12740b6e9d69b3bfb51f13842172efb4fd"}
Oct 30 11:42:36.921620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.921404    5060 scope.go:122] "RemoveContainer" containerID="f5dd357385e1ebbf443f9c2e15edb8e8d9ae205d9af68aed5c04c8ce9c4feacc"
Oct 30 11:42:36.988935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:36.988903    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"]
Oct 30 11:42:37.000547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:37.000512    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-90364398-openshift-default-d8c6b865c-9wjt5"]
Oct 30 11:42:37.351925 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:37.351894    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:37.555759 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:37.555722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:38.333613 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:38.333581    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" path="/var/lib/kubelet/pods/0510329a-49ca-408c-a5ad-1a508dfd86c4/volumes"
Oct 30 11:42:39.046123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:39.046081    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:39.211027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:39.210989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:39.418373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:39.418265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:39.563469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:39.563415    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:40.238231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:40.238197    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"]
Oct 30 11:42:40.379041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:40.379007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:40.521707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:40.521588    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:41.114247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:41.114213    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:41.747104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:41.747069    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:41.889443 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:41.889405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:42.039210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:42.039123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:42.188081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:42.188045    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:42.672169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:42.672137    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:42.974800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:42.974766    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:43.134095 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:43.134053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:44.058553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:44.058518    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:44.255866 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:44.255829    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:45.238819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.238778    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q" podUID="c1a5c283-52e6-4d0b-8941-22c9b95144f2" containerName="payload-pre-processing" containerID="cri-o://6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823" gracePeriod=25
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.424431    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"]
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.425153    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy"
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.425167    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy"
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.425196    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy"
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.425201    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy"
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.425269    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy"
Oct 30 11:42:45.425490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.425281    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0510329a-49ca-408c-a5ad-1a508dfd86c4" containerName="istio-proxy"
Oct 30 11:42:45.428691 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.428672    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.446702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.446678    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"default-dockercfg-l6pc6\""
Oct 30 11:42:45.456842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.456818    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"simulator-w3-4e261b-kserve-self-signed-certs\""
Oct 30 11:42:45.459588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.459480    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"openshift-service-ca.crt\""
Oct 30 11:42:45.459588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.459562    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"kube-root-ca.crt\""
Oct 30 11:42:45.460838 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.460783    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"]
Oct 30 11:42:45.521120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.521091    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:45.522798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.522779    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:42:45.609205 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.609174    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.609385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.609222    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.609385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.609310    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdp4\" (UniqueName: \"kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.609385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.609346    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.609385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.609370    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.609576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.609518    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.709921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.709874    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume\") pod \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") "
Oct 30 11:42:45.710111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710044    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz\" (UniqueName: \"kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz\") pod \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\" (UID: \"c1a5c283-52e6-4d0b-8941-22c9b95144f2\") "
Oct 30 11:42:45.710284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710260    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.710284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710271    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume" pod "c1a5c283-52e6-4d0b-8941-22c9b95144f2" (UID: "c1a5c283-52e6-4d0b-8941-22c9b95144f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:45.710501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710307    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.710501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdp4\" (UniqueName: \"kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.710501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710384    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.710501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710413    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.710718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710508    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.710718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710566    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a5c283-52e6-4d0b-8941-22c9b95144f2-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:45.711342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.710998    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.711342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.711193    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.711342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.711290    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.713189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.713160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.713610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.713589    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.715593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.715566    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz" pod "c1a5c283-52e6-4d0b-8941-22c9b95144f2" (UID: "c1a5c283-52e6-4d0b-8941-22c9b95144f2"). InnerVolumeSpecName "kube-api-access-zwfsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:45.741371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.741345    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdp4\" (UniqueName: \"kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4\") pod \"simulator-w3-4e261b-kserve-686d99bd46-vnl9x\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") " pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.743530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.743515    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:45.812220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:45.812175    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfsz\" (UniqueName: \"kubernetes.io/projected/c1a5c283-52e6-4d0b-8941-22c9b95144f2-kube-api-access-zwfsz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:42:46.021751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.021663    5060 generic.go:356] "Generic (PLEG): container finished" podID="c1a5c283-52e6-4d0b-8941-22c9b95144f2" containerID="6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823" exitCode=0
Oct 30 11:42:46.021751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.021717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q" event={"ID":"c1a5c283-52e6-4d0b-8941-22c9b95144f2","Type":"ContainerDied","Data":"6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823"}
Oct 30 11:42:46.021953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.021781    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"
Oct 30 11:42:46.021953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.021801    5060 scope.go:122] "RemoveContainer" containerID="6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823"
Oct 30 11:42:46.021953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.021788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q" event={"ID":"c1a5c283-52e6-4d0b-8941-22c9b95144f2","Type":"ContainerDied","Data":"c1c16b7770f44752a08ffb2478d7c54fa47700d00b3b00e68883fcd81dd88449"}
Oct 30 11:42:46.032352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.032329    5060 scope.go:122] "RemoveContainer" containerID="6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823"
Oct 30 11:42:46.032791 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:42:46.032768    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823\": container with ID starting with 6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823 not found: ID does not exist" containerID="6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823"
Oct 30 11:42:46.032894 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.032798    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823"} err="failed to get container status \"6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823\": rpc error: code = NotFound desc = could not find container \"6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823\": container with ID starting with 6a03cd3a997e6ccd3aa39c1e05f359a48317623b83ee13b0dd3c4d85629c5823 not found: ID does not exist"
Oct 30 11:42:46.059250 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:42:46.059221    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13d89c0c_fb4e_487e_899e_15edf1495e6e.slice/crio-0b5666bd697dca4e3de9b5d6a75457e15794492983cca820065f86e664d01511 WatchSource:0}: Error finding container 0b5666bd697dca4e3de9b5d6a75457e15794492983cca820065f86e664d01511: Status 404 returned error can't find the container with id 0b5666bd697dca4e3de9b5d6a75457e15794492983cca820065f86e664d01511
Oct 30 11:42:46.059809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.059706    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"]
Oct 30 11:42:46.076960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.076931    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"]
Oct 30 11:42:46.080093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.080072    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64715175-d8659cfcf-8hc7q"]
Oct 30 11:42:46.347744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.346445    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:46.365787 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:46.365143    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a5c283-52e6-4d0b-8941-22c9b95144f2" path="/var/lib/kubelet/pods/c1a5c283-52e6-4d0b-8941-22c9b95144f2/volumes"
Oct 30 11:42:47.030815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:47.030773    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" event={"ID":"13d89c0c-fb4e-487e-899e-15edf1495e6e","Type":"ContainerStarted","Data":"0b5666bd697dca4e3de9b5d6a75457e15794492983cca820065f86e664d01511"}
Oct 30 11:42:47.685074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:47.684972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:48.039971 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:48.039936    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" event={"ID":"13d89c0c-fb4e-487e-899e-15edf1495e6e","Type":"ContainerStarted","Data":"a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e"}
Oct 30 11:42:48.089505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:48.089433    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podStartSLOduration=1.7974545480000002 podStartE2EDuration="3.089419399s" totalImagesPullingTime="1.291964851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:48.08755142 +0000 UTC m=+771.882435914" watchObservedRunningTime="2026-10-30 11:42:48.089419399 +0000 UTC m=+771.884303918"
Oct 30 11:42:48.530919 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:48.530873    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:48.743916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:48.743867    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:49.859132 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:42:49.859091    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:42:50.468910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:50.468867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:52.059975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:52.059924    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:52.918145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:52.918103    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:53.270292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:53.270259    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:53.448214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:53.448177    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:53.637281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:53.637200    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:53.941232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:53.941157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:54.108661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:54.108611    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:54.308080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:54.308015    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:55.163480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:55.163425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:55.366064 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:55.366023    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:55.526870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:55.526834    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:55.744079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:55.744041    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:55.762770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:55.762745    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:56.118269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:56.118233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:56.151243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:56.151213    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:42:56.265262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:56.265226    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:56.411401 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:56.411299    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:56.624355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:56.624315    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:57.234471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.234425    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"]
Oct 30 11:42:57.234936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.234921    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c1a5c283-52e6-4d0b-8941-22c9b95144f2" containerName="payload-pre-processing"
Oct 30 11:42:57.234998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.234937    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a5c283-52e6-4d0b-8941-22c9b95144f2" containerName="payload-pre-processing"
Oct 30 11:42:57.235040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.235010    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a5c283-52e6-4d0b-8941-22c9b95144f2" containerName="payload-pre-processing"
Oct 30 11:42:57.238164 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.238140    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.260838 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.260813    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"kube-root-ca.crt\""
Oct 30 11:42:57.263433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.263394    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"]
Oct 30 11:42:57.265970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.265943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.266118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.266006    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.266118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.266085    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.266224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.266131    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.266224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.266153    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8xg\" (UniqueName: \"kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.266355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.266339    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.271903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.271881    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"openshift-service-ca.crt\""
Oct 30 11:42:57.273295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.273275    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"simulator-w4-849669-kserve-self-signed-certs\""
Oct 30 11:42:57.273395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.273352    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"default-dockercfg-j5w7d\""
Oct 30 11:42:57.341370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.341287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:57.367545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367511    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.367753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367570    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.367753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367603    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8xg\" (UniqueName: \"kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.367753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367724    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.367931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.367931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.368041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.367963    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.368099 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.368040    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.368198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.368144    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.370135 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.370104    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.370517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.370500    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.395048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.394967    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8xg\" (UniqueName: \"kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg\") pod \"simulator-w4-849669-kserve-55b7f65f69-trkn5\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") " pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.559834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.559803    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:57.835581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.835550    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:57.876226 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:42:57.876190    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb84214b8_3b0f_4902_82f2_1cb36c3f2e99.slice/crio-6b23501218c1a55ed8ce88f0968dd47f693be53cae907bf8aec3dfc60267bb39 WatchSource:0}: Error finding container 6b23501218c1a55ed8ce88f0968dd47f693be53cae907bf8aec3dfc60267bb39: Status 404 returned error can't find the container with id 6b23501218c1a55ed8ce88f0968dd47f693be53cae907bf8aec3dfc60267bb39
Oct 30 11:42:57.877268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:57.877240    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"]
Oct 30 11:42:58.017184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.017146    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:58.151136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.151054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" event={"ID":"b84214b8-3b0f-4902-82f2-1cb36c3f2e99","Type":"ContainerStarted","Data":"0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526"}
Oct 30 11:42:58.151136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.151097    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" event={"ID":"b84214b8-3b0f-4902-82f2-1cb36c3f2e99","Type":"ContainerStarted","Data":"6b23501218c1a55ed8ce88f0968dd47f693be53cae907bf8aec3dfc60267bb39"}
Oct 30 11:42:58.152790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.152764    5060 generic.go:356] "Generic (PLEG): container finished" podID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerID="9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102" exitCode=0
Oct 30 11:42:58.152906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.152824    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" event={"ID":"d2248c34-0ee8-4d0a-8b6d-0233e3a29790","Type":"ContainerDied","Data":"9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"}
Oct 30 11:42:58.153148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.153137    5060 scope.go:122] "RemoveContainer" containerID="9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"
Oct 30 11:42:58.154325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.154300    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:58.154393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.154367    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:58.154463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.154396    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:42:58.194887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.194842    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podStartSLOduration=1.194829076 podStartE2EDuration="1.194829076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:58.19276163 +0000 UTC m=+781.987646118" watchObservedRunningTime="2026-10-30 11:42:58.194829076 +0000 UTC m=+781.989713567"
Oct 30 11:42:58.560254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.560217    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:42:58.835016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:58.834909    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:59.161893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:59.161802    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" event={"ID":"d2248c34-0ee8-4d0a-8b6d-0233e3a29790","Type":"ContainerStarted","Data":"f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3"}
Oct 30 11:42:59.470423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:59.470388    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:42:59.471365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:59.471341    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 11:42:59.471500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:59.471397    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 11:42:59.556862 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:59.556829    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:42:59.736838 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:42:59.736749    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:00.470130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:00.470091    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 11:43:00.470323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:00.470148    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 11:43:00.749105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:00.749017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:00.915127 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:00.915083    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:01.450857 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:01.450830    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:01.470590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:01.470559    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 11:43:01.470768 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:01.470624    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 11:43:01.603914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:01.603878    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:01.767126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:01.767030    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:02.470857 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:02.470823    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 11:43:02.471037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:02.470886    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 11:43:02.748194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:02.748094    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:03.471115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:03.471061    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 11:43:03.471708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:03.471143    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 11:43:03.893551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:03.892571    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:43:04.474525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:04.474494    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:43:04.475539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:04.475521    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:43:04.661219 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:04.661179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:06.615887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:06.615848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:06.779105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:06.779067    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:06.930169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:06.930070    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:07.560302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.560265    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:43:07.572943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.572914    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:43:07.766905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.766874    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"]
Oct 30 11:43:07.773574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.773549    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:07.791228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.791197    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"]
Oct 30 11:43:07.791421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.791402    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"simulator-w0-3ccdab-kserve-self-signed-certs\""
Oct 30 11:43:07.791589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.791478    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"openshift-service-ca.crt\""
Oct 30 11:43:07.791636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.791585    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"default-dockercfg-qhshj\""
Oct 30 11:43:07.791771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.791753    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"kube-root-ca.crt\""
Oct 30 11:43:07.853473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.853366    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:07.899598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.899546    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:07.899598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.899604    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:07.899868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.899715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:07.899868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.899806    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:07.899981 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.899871    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:07.900036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:07.899973    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vps4g\" (UniqueName: \"kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.001227 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.001185    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.001421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.001249    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vps4g\" (UniqueName: \"kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.001421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.001335    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.001421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.001366    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.001618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.001427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.001618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.001513    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.002700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.002357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.002700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.002483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.002891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.002800    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.009072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.009047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.025262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.025180    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.028301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.028224    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vps4g\" (UniqueName: \"kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g\") pod \"simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") " pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.086879 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.086841    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:08.250233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.250200    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:43:08.421252 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:43:08.421216    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe201112_afab_4c2c_bb35_b1ea90fc281b.slice/crio-9e67b5dc4fa040a5acf6af1d977e8ad62c0086ca0d5afc0378215006b83762bc WatchSource:0}: Error finding container 9e67b5dc4fa040a5acf6af1d977e8ad62c0086ca0d5afc0378215006b83762bc: Status 404 returned error can't find the container with id 9e67b5dc4fa040a5acf6af1d977e8ad62c0086ca0d5afc0378215006b83762bc
Oct 30 11:43:08.421252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.421223    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"]
Oct 30 11:43:08.553120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:08.553068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:09.247350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.247315    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" event={"ID":"fe201112-afab-4c2c-bb35-b1ea90fc281b","Type":"ContainerStarted","Data":"494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07"}
Oct 30 11:43:09.247886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.247364    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" event={"ID":"fe201112-afab-4c2c-bb35-b1ea90fc281b","Type":"ContainerStarted","Data":"9e67b5dc4fa040a5acf6af1d977e8ad62c0086ca0d5afc0378215006b83762bc"}
Oct 30 11:43:09.291158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.291108    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podStartSLOduration=2.291092565 podStartE2EDuration="2.291092565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:09.289492924 +0000 UTC m=+793.084377414" watchObservedRunningTime="2026-10-30 11:43:09.291092565 +0000 UTC m=+793.085977059"
Oct 30 11:43:09.389928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.389893    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"]
Oct 30 11:43:09.394033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.394012    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.409272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.409241    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"kube-root-ca.crt\""
Oct 30 11:43:09.409591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.409516    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"openshift-service-ca.crt\""
Oct 30 11:43:09.409591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.409554    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"default-dockercfg-sxhks\""
Oct 30 11:43:09.409749 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.409639    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"simulator-w2-1d9b1a-kserve-self-signed-certs\""
Oct 30 11:43:09.413138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.412931    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"]
Oct 30 11:43:09.483297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.483255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:09.516105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.515994    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.516105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.516075    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.516334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.516137    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.516334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.516162    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.516334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.516211    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2gx\" (UniqueName: \"kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.516334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.516264    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.617869 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.617831    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.617896    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.617934    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.617968    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.617985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.618015    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2gx\" (UniqueName: \"kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.618552    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.618905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.618797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.619303 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.619267    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.621930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.621886    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.622082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.622051    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.650504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.650466    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2gx\" (UniqueName: \"kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx\") pod \"simulator-w2-1d9b1a-kserve-667578549c-598p4\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") " pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:09.710093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:09.710050    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:10.021783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.021705    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"]
Oct 30 11:43:10.087174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.087144    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:10.256641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.256593    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" event={"ID":"0ed39ccb-e93a-40af-9214-e566b04e05f7","Type":"ContainerStarted","Data":"456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a"}
Oct 30 11:43:10.256641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.256643    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" event={"ID":"0ed39ccb-e93a-40af-9214-e566b04e05f7","Type":"ContainerStarted","Data":"6d3611182054f4f7bc65e186212e28a6705210a40c4e0d8c08364721448156f0"}
Oct 30 11:43:10.301358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.301252    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podStartSLOduration=1.301238731 podStartE2EDuration="1.301238731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:10.298691887 +0000 UTC m=+794.093576381" watchObservedRunningTime="2026-10-30 11:43:10.301238731 +0000 UTC m=+794.096123224"
Oct 30 11:43:10.446875 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.446834    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:10.710513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:10.710473    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:12.250526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:12.250487    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:14.277560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:14.277520    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:16.126024 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:16.125989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:16.450198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:16.450165    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:18.051916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:18.051865    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:18.087335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:18.087297    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:18.100875 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:18.100846    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:18.315925 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:18.315838    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:43:19.710630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:19.710555    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:19.723788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:19.723760    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:19.856432 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:43:19.856396    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:43:19.880382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:19.880350    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:20.334411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:20.334385    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:43:21.971794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:21.971757    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:22.667539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:22.667499    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:23.321120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:23.321017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:23.509356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:23.509325    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:23.944764 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:23.944724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:26.036499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:26.036409    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:27.429519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.429481    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"]
Oct 30 11:43:27.433515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.433495    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.467018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.466989    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"premium-w4-849669-kserve-self-signed-certs\""
Oct 30 11:43:27.469769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.469742    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"]
Oct 30 11:43:27.509682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.509642    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.509914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.509712    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.509914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.509766    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.509914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.509870    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.509914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.509906    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nsw\" (UniqueName: \"kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.510106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.509987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.513097 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.513071    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:27.611417 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611386    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.611655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611447    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.611655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611496    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nsw\" (UniqueName: \"kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.611655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611542    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.611655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611590    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.611655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611631    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.612020 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.611998    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.612274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.612248    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.612546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.612523    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.615826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.615806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.636801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.636735    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.639000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.638513    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nsw\" (UniqueName: \"kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw\") pod \"premium-w4-849669-kserve-7799bff564-lx9p2\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") " pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:27.745746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:27.745708    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:28.015293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:28.015155    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:28.052907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:28.052865    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"]
Oct 30 11:43:28.053325 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:43:28.053305    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd12ab22d_2ee9_4c17_a3da_7001632db2fa.slice/crio-833925d5fe1654c3df3282b4cb41c4e920770aaecfd3e367d9227a5f6a92a815 WatchSource:0}: Error finding container 833925d5fe1654c3df3282b4cb41c4e920770aaecfd3e367d9227a5f6a92a815: Status 404 returned error can't find the container with id 833925d5fe1654c3df3282b4cb41c4e920770aaecfd3e367d9227a5f6a92a815
Oct 30 11:43:28.374133 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:28.374030    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" event={"ID":"d12ab22d-2ee9-4c17-a3da-7001632db2fa","Type":"ContainerStarted","Data":"9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463"}
Oct 30 11:43:28.374133 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:28.374090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" event={"ID":"d12ab22d-2ee9-4c17-a3da-7001632db2fa","Type":"ContainerStarted","Data":"833925d5fe1654c3df3282b4cb41c4e920770aaecfd3e367d9227a5f6a92a815"}
Oct 30 11:43:28.417352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:28.417301    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podStartSLOduration=1.4172850129999999 podStartE2EDuration="1.417285013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:28.415977091 +0000 UTC m=+812.210861576" watchObservedRunningTime="2026-10-30 11:43:28.417285013 +0000 UTC m=+812.212169506"
Oct 30 11:43:28.746391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:28.746355    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:29.772815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:29.772780    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:31.619757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:31.619718    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:33.542701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:33.542662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:35.108503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:35.108466    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:35.503849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:35.503812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:37.271038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:37.271003    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:37.746578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:37.746541    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:37.759487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:37.759437    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:38.453756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.453728    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:43:38.518736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.518700    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"]
Oct 30 11:43:38.523154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.523126    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.535548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.535504    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"]
Oct 30 11:43:38.550494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.546970    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"premium-w2-1d9b1a-kserve-self-signed-certs\""
Oct 30 11:43:38.627836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.627791    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.628019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.627858    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.628019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.627955    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.628019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.628010    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lz5x\" (UniqueName: \"kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.628195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.628121    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.628195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.628178    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.660939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.660900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:38.729796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.729702    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.729796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.729777    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.729853    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.729899    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.729948    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.730008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lz5x\" (UniqueName: \"kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.730110    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.730312    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.730479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.730425    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.741150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.741117    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.762261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.762230    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.782827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.782792    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lz5x\" (UniqueName: \"kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x\") pod \"premium-w2-1d9b1a-kserve-7584d988d8-tvg54\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") " pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:38.842976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:38.842945    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:39.159635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:39.158470    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"]
Oct 30 11:43:39.186509 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:43:39.186472    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26fdd688_a64c_4ef0_95d5_3e515bdc171c.slice/crio-fdbb7efab8b1dd033e052ba4e478b46646dd2bf3476c6e6ae22ca2988f70fba7 WatchSource:0}: Error finding container fdbb7efab8b1dd033e052ba4e478b46646dd2bf3476c6e6ae22ca2988f70fba7: Status 404 returned error can't find the container with id fdbb7efab8b1dd033e052ba4e478b46646dd2bf3476c6e6ae22ca2988f70fba7
Oct 30 11:43:39.275672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:39.275628    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:39.453126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:39.453083    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" event={"ID":"26fdd688-a64c-4ef0-95d5-3e515bdc171c","Type":"ContainerStarted","Data":"9828724cd2ea77d5d92b5681ecd7b2619da3519005ad16ffb9d0c6d16ddadb9f"}
Oct 30 11:43:39.453126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:39.453127    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" event={"ID":"26fdd688-a64c-4ef0-95d5-3e515bdc171c","Type":"ContainerStarted","Data":"fdbb7efab8b1dd033e052ba4e478b46646dd2bf3476c6e6ae22ca2988f70fba7"}
Oct 30 11:43:39.496321 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:39.496237    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podStartSLOduration=1.496219362 podStartE2EDuration="1.496219362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:39.494886146 +0000 UTC m=+823.289770641" watchObservedRunningTime="2026-10-30 11:43:39.496219362 +0000 UTC m=+823.291103857"
Oct 30 11:43:39.843851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:39.843757    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:40.107530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:40.103815    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:41.204033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:41.203994    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:43.072386 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:43.072336    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:44.965303 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:44.965262    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:46.770328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:46.770290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:47.844826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:47.844785    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:48.779501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.779422    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:48.843773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.843744    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:48.858126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.858094    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:48.860677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.860646    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"]
Oct 30 11:43:48.865585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.865563    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.887289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.887261    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"]
Oct 30 11:43:48.888309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.888285    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4e261b-openshift-default-dockercfg-slt8g\""
Oct 30 11:43:48.938711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.938674    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.938711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.938709    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.938972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.938749    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.938972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.938873    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.939085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.939014    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.939143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.939122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.939214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.939197    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.939286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.939273    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc998\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:48.939350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:48.939327    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040423    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040511    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040555    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc998\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040589    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040643    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040734    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040769    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.040966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.040792    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.041360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.041253    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.041813 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.041789    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.042129 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.042102    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.042254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.042228    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.042562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.042535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.043624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.043605    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.043998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.043976    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.064535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.064499    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.064700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.064622    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc998\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.181827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.181791    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:49.515626 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:43:49.515575    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed8feed_bf08_429d_9a7c_bd8cadbee54e.slice/crio-2e54ef011d79665dc9847b1a8bb0de06e1108eca7f8b7b85819f85a09b976328 WatchSource:0}: Error finding container 2e54ef011d79665dc9847b1a8bb0de06e1108eca7f8b7b85819f85a09b976328: Status 404 returned error can't find the container with id 2e54ef011d79665dc9847b1a8bb0de06e1108eca7f8b7b85819f85a09b976328
Oct 30 11:43:49.516675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.516641    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"]
Oct 30 11:43:49.518687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.518650    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:43:49.518803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.518731    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:43:49.518803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.518770    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:43:49.535524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:49.535488    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:43:49.860551 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:43:49.859488    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:43:50.528155 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:50.528119    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" event={"ID":"0ed8feed-bf08-429d-9a7c-bd8cadbee54e","Type":"ContainerStarted","Data":"63bfa5af53e8d15d473ab190e63bb2a86897040b6770cac65aa01efebe805f5b"}
Oct 30 11:43:50.528378 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:50.528180    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" event={"ID":"0ed8feed-bf08-429d-9a7c-bd8cadbee54e","Type":"ContainerStarted","Data":"2e54ef011d79665dc9847b1a8bb0de06e1108eca7f8b7b85819f85a09b976328"}
Oct 30 11:43:50.774991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:50.774937    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:51.186555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:51.186519    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:51.187193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:51.187020    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:43:51.187358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:51.187335    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 11:43:52.185907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:52.185866    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:43:52.186249 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:52.186218    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 11:43:52.618107 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:52.618066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:53.186481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:53.183851    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:43:53.186481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:53.183916    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 11:43:54.184480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:54.182533    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:43:54.184480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:54.182593    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 11:43:54.502236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:54.502198    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:55.187921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:55.187879    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:43:55.188608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:55.188564    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 11:43:56.309582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:56.309527    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:56.310348 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:56.310317    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:56.311220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:56.311203    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:43:56.380033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:56.379975    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podStartSLOduration=8.379957013 podStartE2EDuration="8.379957013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:50.597470077 +0000 UTC m=+834.392354568" watchObservedRunningTime="2026-10-30 11:43:56.379957013 +0000 UTC m=+840.174841509"
Oct 30 11:43:56.496370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:56.496325    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:43:58.491064 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:43:58.491017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:00.374679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:00.374607    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:02.271586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:02.271545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:04.106785 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:04.106747    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:06.153939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:06.153899    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:07.238648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:07.238603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:08.193079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:08.193045    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:09.995816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:09.995778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:11.859593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:11.859556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:13.773940 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:13.773901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:15.548324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:15.548287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:17.673203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:17.673164    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:19.859379 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:44:19.859338    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:44:19.872252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:19.872223    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:21.778288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:21.778247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:23.659301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:23.659253    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:25.577026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:25.576965    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:27.492597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:27.492556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:29.266959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:29.266925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:31.094795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:31.094765    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:32.901614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:32.898900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:34.824441 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:34.824405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:37.015613 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:37.015571    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:38.694825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.694790    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"]
Oct 30 11:44:38.698363 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.698339    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.713468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.713436    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e43a66-openshift-default-dockercfg-pvppc\""
Oct 30 11:44:38.728721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.728687    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"]
Oct 30 11:44:38.866351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.866308    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:38.879283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879369    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879398    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879536    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nt4\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879580    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879661    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.879897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.879726    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.980964 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.980932    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nt4\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.980975    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981112    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981155    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981311 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981362    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981388    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981514    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981761 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981736    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981837    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.981907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.981890    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.983944 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.983926    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:38.984026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:38.984006    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:39.005019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.004996    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nt4\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:39.005308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.005291    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:39.011582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.011566    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:39.338111 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:44:39.338065    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde97e275_4658_4d9e_b5a9_ccf8c52321d4.slice/crio-b8da070b8f11644bee58bd4d1ef5a37939a0bdb323847e63da511fd4c8a6d1b0 WatchSource:0}: Error finding container b8da070b8f11644bee58bd4d1ef5a37939a0bdb323847e63da511fd4c8a6d1b0: Status 404 returned error can't find the container with id b8da070b8f11644bee58bd4d1ef5a37939a0bdb323847e63da511fd4c8a6d1b0
Oct 30 11:44:39.339046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.339019    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"]
Oct 30 11:44:39.340744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.340706    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:44:39.340840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.340788    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:44:39.340840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.340832    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:44:39.851469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.851428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" event={"ID":"de97e275-4658-4d9e-b5a9-ccf8c52321d4","Type":"ContainerStarted","Data":"23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6"}
Oct 30 11:44:39.852005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.851503    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" event={"ID":"de97e275-4658-4d9e-b5a9-ccf8c52321d4","Type":"ContainerStarted","Data":"b8da070b8f11644bee58bd4d1ef5a37939a0bdb323847e63da511fd4c8a6d1b0"}
Oct 30 11:44:39.940388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:39.940338    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podStartSLOduration=1.9403237020000002 podStartE2EDuration="1.940323702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:39.92629881 +0000 UTC m=+883.721183303" watchObservedRunningTime="2026-10-30 11:44:39.940323702 +0000 UTC m=+883.735208195"
Oct 30 11:44:40.012061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:40.012032    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:40.014006 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:40.013977    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:44:40.014167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:40.014029    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 11:44:40.818487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:40.816990    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:41.013077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:41.013041    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:44:41.013531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:41.013105    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 11:44:42.013034 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:42.012988    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:44:42.013262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:42.013047    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 11:44:42.705159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:42.705079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:43.013092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:43.012985    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:44:43.013092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:43.013058    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 11:44:44.012235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:44.012202    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:44:44.012414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:44.012255    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 11:44:44.590523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:44.590488    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:45.012313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:45.012285    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:44:45.012566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:45.012545    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 11:44:46.029669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:46.029631    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:46.030529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:46.030506    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:46.031254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:46.031232    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 11:44:46.367031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:46.366932    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:47.958600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:47.958549    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:48.269081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:48.268975    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:49.857346 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:44:49.857310    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:44:50.169962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:50.169863    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:52.169120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:52.169087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:54.330414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:54.330375    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:56.319056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:56.319021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:56.447128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:56.447097    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:44:56.449738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:56.449715    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:44:58.184967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:58.184926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:44:59.985984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:44:59.985935    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:00.175930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.175882    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"]
Oct 30 11:45:00.180107 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.180077    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"]
Oct 30 11:45:00.180299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.180203    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.183746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.183722    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"]
Oct 30 11:45:00.183897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.183853    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.187279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.187257    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.196526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.196496    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"]
Oct 30 11:45:00.206096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.206065    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:45:00.206278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.206246    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:45:00.206354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.206336    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:45:00.209327 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.209302    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"]
Oct 30 11:45:00.215203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.215176    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"]
Oct 30 11:45:00.349602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.349494    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cjj\" (UniqueName: \"kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.349602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.349553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.349818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.349695    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.349818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.349728    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.349818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.349746    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42ng\" (UniqueName: \"kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.349931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.349868    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.450909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.450873    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.451121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.450934    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cjj\" (UniqueName: \"kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.451121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.450959    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.451121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.450992    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.451121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.451010    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.451342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.451143    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42ng\" (UniqueName: \"kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.451722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.451702    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.451804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.451752    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.451804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.451760    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.475347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.475309    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cjj\" (UniqueName: \"kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.475565 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.475540    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42ng\" (UniqueName: \"kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.475702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.475677    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.496113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.496083    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:00.523008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.522966    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:00.529476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.529430    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:00.823959 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:45:00.823886    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7540ff_7710_4e58_9ac0_138720256e40.slice/crio-bd5d44554eba860c737ae6f374af24372f3738511891e67542fa3325698719b9 WatchSource:0}: Error finding container bd5d44554eba860c737ae6f374af24372f3738511891e67542fa3325698719b9: Status 404 returned error can't find the container with id bd5d44554eba860c737ae6f374af24372f3738511891e67542fa3325698719b9
Oct 30 11:45:00.824844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.824817    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"]
Oct 30 11:45:00.907224 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:45:00.907191    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ad2222_ce50_45ed_a431_e4e269033045.slice/crio-7f6f412bd896ad62af4779ade090f4112a2f955343e4462ab115edfce7ab0ae8 WatchSource:0}: Error finding container 7f6f412bd896ad62af4779ade090f4112a2f955343e4462ab115edfce7ab0ae8: Status 404 returned error can't find the container with id 7f6f412bd896ad62af4779ade090f4112a2f955343e4462ab115edfce7ab0ae8
Oct 30 11:45:00.907604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.907582    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"]
Oct 30 11:45:00.910497 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:45:00.910466    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ce31d9_5fb5_40aa_9a1f_e719f7774a39.slice/crio-8fa29de7193049ae7938440d08bbed2eb21d9afc4b55f94e9fbe8b6f760e799f WatchSource:0}: Error finding container 8fa29de7193049ae7938440d08bbed2eb21d9afc4b55f94e9fbe8b6f760e799f: Status 404 returned error can't find the container with id 8fa29de7193049ae7938440d08bbed2eb21d9afc4b55f94e9fbe8b6f760e799f
Oct 30 11:45:00.910757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:00.910736    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"]
Oct 30 11:45:01.011890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:01.011847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" event={"ID":"3b7540ff-7710-4e58-9ac0-138720256e40","Type":"ContainerStarted","Data":"bd5d44554eba860c737ae6f374af24372f3738511891e67542fa3325698719b9"}
Oct 30 11:45:01.013125 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:01.013084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" event={"ID":"68ad2222-ce50-45ed-a431-e4e269033045","Type":"ContainerStarted","Data":"7f6f412bd896ad62af4779ade090f4112a2f955343e4462ab115edfce7ab0ae8"}
Oct 30 11:45:01.014481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:01.014428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" event={"ID":"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39","Type":"ContainerStarted","Data":"8fa29de7193049ae7938440d08bbed2eb21d9afc4b55f94e9fbe8b6f760e799f"}
Oct 30 11:45:01.963804 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:01.963637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:02.023983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.023949    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b7540ff-7710-4e58-9ac0-138720256e40" containerID="72aad777647b72dedcc1b600c57b798acbd7aa7d71ffdc5dc99b91463e4c1fff" exitCode=60
Oct 30 11:45:02.024540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.024031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" event={"ID":"3b7540ff-7710-4e58-9ac0-138720256e40","Type":"ContainerDied","Data":"72aad777647b72dedcc1b600c57b798acbd7aa7d71ffdc5dc99b91463e4c1fff"}
Oct 30 11:45:02.024540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.024352    5060 scope.go:122] "RemoveContainer" containerID="72aad777647b72dedcc1b600c57b798acbd7aa7d71ffdc5dc99b91463e4c1fff"
Oct 30 11:45:02.026287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.026268    5060 generic.go:356] "Generic (PLEG): container finished" podID="68ad2222-ce50-45ed-a431-e4e269033045" containerID="a459a4169ec5bd13f6b5ae80a28395f9b76278fc2642d873ca3a9b16e560a163" exitCode=60
Oct 30 11:45:02.026364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.026344    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" event={"ID":"68ad2222-ce50-45ed-a431-e4e269033045","Type":"ContainerDied","Data":"a459a4169ec5bd13f6b5ae80a28395f9b76278fc2642d873ca3a9b16e560a163"}
Oct 30 11:45:02.026639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.026618    5060 scope.go:122] "RemoveContainer" containerID="a459a4169ec5bd13f6b5ae80a28395f9b76278fc2642d873ca3a9b16e560a163"
Oct 30 11:45:02.028357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.028333    5060 generic.go:356] "Generic (PLEG): container finished" podID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerID="b4ca6fbdaa87aa9c9968781ae80736f62c9e95c3fff3c6db5140fe513573f205" exitCode=60
Oct 30 11:45:02.028490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.028412    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" event={"ID":"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39","Type":"ContainerDied","Data":"b4ca6fbdaa87aa9c9968781ae80736f62c9e95c3fff3c6db5140fe513573f205"}
Oct 30 11:45:02.028663 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:02.028650    5060 scope.go:122] "RemoveContainer" containerID="b4ca6fbdaa87aa9c9968781ae80736f62c9e95c3fff3c6db5140fe513573f205"
Oct 30 11:45:03.041208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.040680    5060 generic.go:356] "Generic (PLEG): container finished" podID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerID="d54df25c9038ed54c05f331ada66e768d70984629a98cf00797870bd333ac7f9" exitCode=60
Oct 30 11:45:03.041208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.040896    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" event={"ID":"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39","Type":"ContainerDied","Data":"d54df25c9038ed54c05f331ada66e768d70984629a98cf00797870bd333ac7f9"}
Oct 30 11:45:03.041208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.040951    5060 scope.go:122] "RemoveContainer" containerID="b4ca6fbdaa87aa9c9968781ae80736f62c9e95c3fff3c6db5140fe513573f205"
Oct 30 11:45:03.041741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.041436    5060 scope.go:122] "RemoveContainer" containerID="d54df25c9038ed54c05f331ada66e768d70984629a98cf00797870bd333ac7f9"
Oct 30 11:45:03.041797 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:03.041765    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl_odh-ai-gateway-infra(b0ce31d9-5fb5-40aa-9a1f-e719f7774a39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39"
Oct 30 11:45:03.044240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.044217    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b7540ff-7710-4e58-9ac0-138720256e40" containerID="6bdb85234afbd17c8ab811d49b26c78d9b81bb2c95571ba2bff21162e396c094" exitCode=60
Oct 30 11:45:03.044373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.044298    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" event={"ID":"3b7540ff-7710-4e58-9ac0-138720256e40","Type":"ContainerDied","Data":"6bdb85234afbd17c8ab811d49b26c78d9b81bb2c95571ba2bff21162e396c094"}
Oct 30 11:45:03.044587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.044572    5060 scope.go:122] "RemoveContainer" containerID="6bdb85234afbd17c8ab811d49b26c78d9b81bb2c95571ba2bff21162e396c094"
Oct 30 11:45:03.044867 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:03.044837    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln_odh-ai-gateway-infra(3b7540ff-7710-4e58-9ac0-138720256e40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" podUID="3b7540ff-7710-4e58-9ac0-138720256e40"
Oct 30 11:45:03.046169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.046146    5060 generic.go:356] "Generic (PLEG): container finished" podID="68ad2222-ce50-45ed-a431-e4e269033045" containerID="ef631652e641463eae570bad16c8f86bf5f13e933fce9b52c802f2cbd9afd034" exitCode=60
Oct 30 11:45:03.046240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.046222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" event={"ID":"68ad2222-ce50-45ed-a431-e4e269033045","Type":"ContainerDied","Data":"ef631652e641463eae570bad16c8f86bf5f13e933fce9b52c802f2cbd9afd034"}
Oct 30 11:45:03.046424 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.046410    5060 scope.go:122] "RemoveContainer" containerID="ef631652e641463eae570bad16c8f86bf5f13e933fce9b52c802f2cbd9afd034"
Oct 30 11:45:03.046683 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:03.046662    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz_odh-ai-gateway-infra(68ad2222-ce50-45ed-a431-e4e269033045)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" podUID="68ad2222-ce50-45ed-a431-e4e269033045"
Oct 30 11:45:03.156138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.156107    5060 scope.go:122] "RemoveContainer" containerID="72aad777647b72dedcc1b600c57b798acbd7aa7d71ffdc5dc99b91463e4c1fff"
Oct 30 11:45:03.169878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.169850    5060 scope.go:122] "RemoveContainer" containerID="a459a4169ec5bd13f6b5ae80a28395f9b76278fc2642d873ca3a9b16e560a163"
Oct 30 11:45:03.889131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:03.889093    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:04.055191 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:04.055166    5060 scope.go:122] "RemoveContainer" containerID="ef631652e641463eae570bad16c8f86bf5f13e933fce9b52c802f2cbd9afd034"
Oct 30 11:45:04.055680 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:04.055443    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz_odh-ai-gateway-infra(68ad2222-ce50-45ed-a431-e4e269033045)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" podUID="68ad2222-ce50-45ed-a431-e4e269033045"
Oct 30 11:45:04.056919 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:04.056898    5060 scope.go:122] "RemoveContainer" containerID="d54df25c9038ed54c05f331ada66e768d70984629a98cf00797870bd333ac7f9"
Oct 30 11:45:04.057167 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:04.057144    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl_odh-ai-gateway-infra(b0ce31d9-5fb5-40aa-9a1f-e719f7774a39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39"
Oct 30 11:45:04.058542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:04.058525    5060 scope.go:122] "RemoveContainer" containerID="6bdb85234afbd17c8ab811d49b26c78d9b81bb2c95571ba2bff21162e396c094"
Oct 30 11:45:04.058746 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:04.058731    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln_odh-ai-gateway-infra(3b7540ff-7710-4e58-9ac0-138720256e40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" podUID="3b7540ff-7710-4e58-9ac0-138720256e40"
Oct 30 11:45:05.684182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:05.684145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:05.953266 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:05.953232    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"]
Oct 30 11:45:05.958388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:05.958369    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:05.999495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:05.999464    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:45:06.004192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.004160    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"]
Oct 30 11:45:06.028328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.028294    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.028508 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.028336    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqwm\" (UniqueName: \"kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.028508 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.028436    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.129889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.129853    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.129889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.129898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqwm\" (UniqueName: \"kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.130116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.129944    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.130558 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.130535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.132548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.132525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.410889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.410808    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqwm\" (UniqueName: \"kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm\") pod \"maas-api-revoke-sub-4bae4b2f-l5wd9\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:06.571834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:06.571796    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:07.158151 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:45:07.158120    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6273511_94f4_405f_a89d_e76ddcbfd903.slice/crio-28fb37cb3516fd7d99189f1f5e2e3c898037eb57dc926bac4772c1ee632c416d WatchSource:0}: Error finding container 28fb37cb3516fd7d99189f1f5e2e3c898037eb57dc926bac4772c1ee632c416d: Status 404 returned error can't find the container with id 28fb37cb3516fd7d99189f1f5e2e3c898037eb57dc926bac4772c1ee632c416d
Oct 30 11:45:07.158852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:07.158824    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"]
Oct 30 11:45:08.086192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:08.086155    5060 generic.go:356] "Generic (PLEG): container finished" podID="e6273511-94f4-405f-a89d-e76ddcbfd903" containerID="901a2414d11166f01e9d35dfe36d42891911e4e9b873f189e5969cfcee97d127" exitCode=0
Oct 30 11:45:08.086374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:08.086239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9" event={"ID":"e6273511-94f4-405f-a89d-e76ddcbfd903","Type":"ContainerDied","Data":"901a2414d11166f01e9d35dfe36d42891911e4e9b873f189e5969cfcee97d127"}
Oct 30 11:45:08.086374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:08.086338    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9" event={"ID":"e6273511-94f4-405f-a89d-e76ddcbfd903","Type":"ContainerStarted","Data":"28fb37cb3516fd7d99189f1f5e2e3c898037eb57dc926bac4772c1ee632c416d"}
Oct 30 11:45:08.765428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:08.765388    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:09.253781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.253756    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:09.368012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.367932    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca\") pod \"e6273511-94f4-405f-a89d-e76ddcbfd903\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") "
Oct 30 11:45:09.368012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.367990    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca\") pod \"e6273511-94f4-405f-a89d-e76ddcbfd903\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") "
Oct 30 11:45:09.368207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.368119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm\" (UniqueName: \"kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm\") pod \"e6273511-94f4-405f-a89d-e76ddcbfd903\" (UID: \"e6273511-94f4-405f-a89d-e76ddcbfd903\") "
Oct 30 11:45:09.368363 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.368337    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca" pod "e6273511-94f4-405f-a89d-e76ddcbfd903" (UID: "e6273511-94f4-405f-a89d-e76ddcbfd903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:45:09.368630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.368600    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:09.370252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.370226    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca" pod "e6273511-94f4-405f-a89d-e76ddcbfd903" (UID: "e6273511-94f4-405f-a89d-e76ddcbfd903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:45:09.370323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.370305    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm" pod "e6273511-94f4-405f-a89d-e76ddcbfd903" (UID: "e6273511-94f4-405f-a89d-e76ddcbfd903"). InnerVolumeSpecName "kube-api-access-7kqwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:45:09.469390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.469349    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqwm\" (UniqueName: \"kubernetes.io/projected/e6273511-94f4-405f-a89d-e76ddcbfd903-kube-api-access-7kqwm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:09.469390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:09.469391    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6273511-94f4-405f-a89d-e76ddcbfd903-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:10.102304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:10.102268    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9" event={"ID":"e6273511-94f4-405f-a89d-e76ddcbfd903","Type":"ContainerDied","Data":"28fb37cb3516fd7d99189f1f5e2e3c898037eb57dc926bac4772c1ee632c416d"}
Oct 30 11:45:10.102852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:10.102328    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28fb37cb3516fd7d99189f1f5e2e3c898037eb57dc926bac4772c1ee632c416d"
Oct 30 11:45:10.102852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:10.102275    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"
Oct 30 11:45:10.683751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:10.683718    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:12.031750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:12.031704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:12.587796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:12.587756    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:13.327355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:13.327319    5060 scope.go:122] "RemoveContainer" containerID="d54df25c9038ed54c05f331ada66e768d70984629a98cf00797870bd333ac7f9"
Oct 30 11:45:14.129598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:14.129566    5060 generic.go:356] "Generic (PLEG): container finished" podID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerID="540bd59073cc3c8ac6496e4dd79b658c846d6a80b672e069a8373425f0d6a75e" exitCode=60
Oct 30 11:45:14.129788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:14.129645    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" event={"ID":"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39","Type":"ContainerDied","Data":"540bd59073cc3c8ac6496e4dd79b658c846d6a80b672e069a8373425f0d6a75e"}
Oct 30 11:45:14.129788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:14.129704    5060 scope.go:122] "RemoveContainer" containerID="d54df25c9038ed54c05f331ada66e768d70984629a98cf00797870bd333ac7f9"
Oct 30 11:45:14.130164 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:14.130114    5060 scope.go:122] "RemoveContainer" containerID="540bd59073cc3c8ac6496e4dd79b658c846d6a80b672e069a8373425f0d6a75e"
Oct 30 11:45:14.130445 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:14.130411    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl_odh-ai-gateway-infra(b0ce31d9-5fb5-40aa-9a1f-e719f7774a39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39"
Oct 30 11:45:14.327251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:14.327219    5060 scope.go:122] "RemoveContainer" containerID="ef631652e641463eae570bad16c8f86bf5f13e933fce9b52c802f2cbd9afd034"
Oct 30 11:45:14.471307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:14.471159    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:15.137604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.137574    5060 generic.go:356] "Generic (PLEG): container finished" podID="68ad2222-ce50-45ed-a431-e4e269033045" containerID="f2e04fce27bedfa3112b417cd06577fe8093c2167a1c0aeac017344ff7ed90e4" exitCode=60
Oct 30 11:45:15.137811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.137648    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" event={"ID":"68ad2222-ce50-45ed-a431-e4e269033045","Type":"ContainerDied","Data":"f2e04fce27bedfa3112b417cd06577fe8093c2167a1c0aeac017344ff7ed90e4"}
Oct 30 11:45:15.137811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.137711    5060 scope.go:122] "RemoveContainer" containerID="ef631652e641463eae570bad16c8f86bf5f13e933fce9b52c802f2cbd9afd034"
Oct 30 11:45:15.138092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.138077    5060 scope.go:122] "RemoveContainer" containerID="f2e04fce27bedfa3112b417cd06577fe8093c2167a1c0aeac017344ff7ed90e4"
Oct 30 11:45:15.138352 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:15.138334    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz_odh-ai-gateway-infra(68ad2222-ce50-45ed-a431-e4e269033045)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" podUID="68ad2222-ce50-45ed-a431-e4e269033045"
Oct 30 11:45:15.196773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.196733    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"]
Oct 30 11:45:15.327947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.327917    5060 scope.go:122] "RemoveContainer" containerID="6bdb85234afbd17c8ab811d49b26c78d9b81bb2c95571ba2bff21162e396c094"
Oct 30 11:45:15.388900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.388870    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:15.435795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.435761    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq\") pod \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") "
Oct 30 11:45:15.436015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.435848    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca\") pod \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\" (UID: \"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39\") "
Oct 30 11:45:15.436952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.436523    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca" pod "b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" (UID: "b0ce31d9-5fb5-40aa-9a1f-e719f7774a39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:45:15.439395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.439361    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq" pod "b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" (UID: "b0ce31d9-5fb5-40aa-9a1f-e719f7774a39"). InnerVolumeSpecName "kube-api-access-sbvmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:45:15.537492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.537402    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:15.537492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:15.537466    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39-kube-api-access-sbvmq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:16.147478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.147386    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"
Oct 30 11:45:16.147478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.147425    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl" event={"ID":"b0ce31d9-5fb5-40aa-9a1f-e719f7774a39","Type":"ContainerDied","Data":"8fa29de7193049ae7938440d08bbed2eb21d9afc4b55f94e9fbe8b6f760e799f"}
Oct 30 11:45:16.147692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.147510    5060 scope.go:122] "RemoveContainer" containerID="540bd59073cc3c8ac6496e4dd79b658c846d6a80b672e069a8373425f0d6a75e"
Oct 30 11:45:16.149959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.149927    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" event={"ID":"3b7540ff-7710-4e58-9ac0-138720256e40","Type":"ContainerStarted","Data":"d7bb0d2b9c2fd379f39185e6e100d735f396e6249eb0138661560b2ab607e0c4"}
Oct 30 11:45:16.199005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.198966    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"]
Oct 30 11:45:16.203535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.203501    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"]
Oct 30 11:45:16.214136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.214107    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889345-4clbl"]
Oct 30 11:45:16.263987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.263934    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" podStartSLOduration=16.263911097 podStartE2EDuration="16.263911097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:16.262437603 +0000 UTC m=+920.057322090" watchObservedRunningTime="2026-10-30 11:45:16.263911097 +0000 UTC m=+920.058795590"
Oct 30 11:45:16.351382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.351349    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" path="/var/lib/kubelet/pods/b0ce31d9-5fb5-40aa-9a1f-e719f7774a39/volumes"
Oct 30 11:45:16.386630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.386602    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:16.406629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.406439    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:16.447237 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.447208    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng\" (UniqueName: \"kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng\") pod \"68ad2222-ce50-45ed-a431-e4e269033045\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") "
Oct 30 11:45:16.447414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.447394    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca\") pod \"68ad2222-ce50-45ed-a431-e4e269033045\" (UID: \"68ad2222-ce50-45ed-a431-e4e269033045\") "
Oct 30 11:45:16.447836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.447810    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca" pod "68ad2222-ce50-45ed-a431-e4e269033045" (UID: "68ad2222-ce50-45ed-a431-e4e269033045"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:45:16.449659 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.449635    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng" pod "68ad2222-ce50-45ed-a431-e4e269033045" (UID: "68ad2222-ce50-45ed-a431-e4e269033045"). InnerVolumeSpecName "kube-api-access-s42ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:45:16.549316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.549273    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42ng\" (UniqueName: \"kubernetes.io/projected/68ad2222-ce50-45ed-a431-e4e269033045-kube-api-access-s42ng\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:16.549316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:16.549310    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ad2222-ce50-45ed-a431-e4e269033045-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:17.164678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.164645    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b7540ff-7710-4e58-9ac0-138720256e40" containerID="d7bb0d2b9c2fd379f39185e6e100d735f396e6249eb0138661560b2ab607e0c4" exitCode=60
Oct 30 11:45:17.164886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.164709    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" event={"ID":"3b7540ff-7710-4e58-9ac0-138720256e40","Type":"ContainerDied","Data":"d7bb0d2b9c2fd379f39185e6e100d735f396e6249eb0138661560b2ab607e0c4"}
Oct 30 11:45:17.164886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.164779    5060 scope.go:122] "RemoveContainer" containerID="6bdb85234afbd17c8ab811d49b26c78d9b81bb2c95571ba2bff21162e396c094"
Oct 30 11:45:17.165154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.165131    5060 scope.go:122] "RemoveContainer" containerID="d7bb0d2b9c2fd379f39185e6e100d735f396e6249eb0138661560b2ab607e0c4"
Oct 30 11:45:17.165521 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:17.165435    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln_odh-ai-gateway-infra(3b7540ff-7710-4e58-9ac0-138720256e40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" podUID="3b7540ff-7710-4e58-9ac0-138720256e40"
Oct 30 11:45:17.166471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.166428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz" event={"ID":"68ad2222-ce50-45ed-a431-e4e269033045","Type":"ContainerDied","Data":"7f6f412bd896ad62af4779ade090f4112a2f955343e4462ab115edfce7ab0ae8"}
Oct 30 11:45:17.166553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.166539    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"
Oct 30 11:45:17.181427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.181408    5060 scope.go:122] "RemoveContainer" containerID="f2e04fce27bedfa3112b417cd06577fe8093c2167a1c0aeac017344ff7ed90e4"
Oct 30 11:45:17.280564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.280514    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"]
Oct 30 11:45:17.355809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.355773    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"]
Oct 30 11:45:17.362597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:17.362566    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889345-gxzxz"]
Oct 30 11:45:18.272584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.272544    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:18.334609 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.334577    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ad2222-ce50-45ed-a431-e4e269033045" path="/var/lib/kubelet/pods/68ad2222-ce50-45ed-a431-e4e269033045/volumes"
Oct 30 11:45:18.350446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.350418    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:18.472599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.472571    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca\") pod \"3b7540ff-7710-4e58-9ac0-138720256e40\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") "
Oct 30 11:45:18.472808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.472653    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj\" (UniqueName: \"kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj\") pod \"3b7540ff-7710-4e58-9ac0-138720256e40\" (UID: \"3b7540ff-7710-4e58-9ac0-138720256e40\") "
Oct 30 11:45:18.473035 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.473007    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca" pod "3b7540ff-7710-4e58-9ac0-138720256e40" (UID: "3b7540ff-7710-4e58-9ac0-138720256e40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:45:18.475194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.475174    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj" pod "3b7540ff-7710-4e58-9ac0-138720256e40" (UID: "3b7540ff-7710-4e58-9ac0-138720256e40"). InnerVolumeSpecName "kube-api-access-h8cjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:45:18.573849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.573815    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7540ff-7710-4e58-9ac0-138720256e40-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:18.573849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:18.573845    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cjj\" (UniqueName: \"kubernetes.io/projected/3b7540ff-7710-4e58-9ac0-138720256e40-kube-api-access-h8cjj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:45:19.185779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:19.185744    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"
Oct 30 11:45:19.185779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:19.185764    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln" event={"ID":"3b7540ff-7710-4e58-9ac0-138720256e40","Type":"ContainerDied","Data":"bd5d44554eba860c737ae6f374af24372f3738511891e67542fa3325698719b9"}
Oct 30 11:45:19.186039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:19.185837    5060 scope.go:122] "RemoveContainer" containerID="d7bb0d2b9c2fd379f39185e6e100d735f396e6249eb0138661560b2ab607e0c4"
Oct 30 11:45:19.238731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:19.238695    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"]
Oct 30 11:45:19.251646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:19.251610    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889345-97rln"]
Oct 30 11:45:19.856901 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:19.856865    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:45:20.226062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:20.226028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:20.335175 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:20.335138    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" path="/var/lib/kubelet/pods/3b7540ff-7710-4e58-9ac0-138720256e40/volumes"
Oct 30 11:45:22.099034 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:22.099000    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:23.858780 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:23.858746    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:25.821067 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:25.821028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:27.704977 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:27.704938    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:29.478566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:29.478527    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:29.659284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:29.659248    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:31.508792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:31.508746    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:33.530000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:33.529963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:35.465141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:35.465095    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:37.275776 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:37.275687    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:39.375551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:39.375514    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:41.147171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:41.147135    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:43.084976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:43.084928    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:44.892145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:44.892085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:46.868984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:46.868945    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:47.988800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:47.988765    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:48.198927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:48.198881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:48.722962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:48.722927    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:49.857088 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:45:49.857056    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:45:50.624810 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:50.624774    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:52.293710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:52.293672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:52.444530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:52.444496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:54.240066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:54.240031    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:55.960035 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:55.959996    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:57.792546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:57.792505    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:45:59.740549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:45:59.740505    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:01.553313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:01.553267    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:03.367686 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:03.367651    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:04.686035 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:04.685998    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:05.356742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:05.356706    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:07.328060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:07.328019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:09.285824 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:09.285775    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:11.068049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:11.067999    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:12.944848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:12.944810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:14.500434 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:14.500400    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:14.659578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:14.659548    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:15.810974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:15.810942    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:16.458844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:16.458796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:19.318208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318171    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"]
Oct 30 11:46:19.318934 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318911    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.318934 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318934    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318951    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318961    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318980    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318988    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.318998    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319005    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319016    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319023    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319035    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319042    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319051    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319058    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319080    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e6273511-94f4-405f-a89d-e76ddcbfd903" containerName="revoke-keys"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319086    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6273511-94f4-405f-a89d-e76ddcbfd903" containerName="revoke-keys"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319103    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:19.319111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319110    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319235    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319246    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319256    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319264    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319271    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319282    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6273511-94f4-405f-a89d-e76ddcbfd903" containerName="revoke-keys"
Oct 30 11:46:19.319593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.319293    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:19.323413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.323379    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.344488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.344441    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"]
Oct 30 11:46:19.426583 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.426540    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.426789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.426603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.426789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.426653    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np72w\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.426789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.426770    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.426958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.426807    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.427016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.426972    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.427151 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.427132    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.427216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.427184    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.427216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.427204    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.528853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528815    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528860    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528881    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528916    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528954    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np72w\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.528986    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.529005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.529044    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.529481    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.529560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529729 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.529699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.529911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.529759    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.530153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.530129    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.531883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.531861    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.532110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.532092    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.553232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.553199    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.553378 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.553338    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np72w\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.638232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.638150    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:19.855840 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:46:19.855809    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:46:19.957875 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:46:19.957839    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a37abf_a6cc_4e03_8490_e1c9da962c87.slice/crio-8c14318ba753c7f274be6e567ff7991e6eb98b9ac4f4b4a4d6ad7748cc23bab5 WatchSource:0}: Error finding container 8c14318ba753c7f274be6e567ff7991e6eb98b9ac4f4b4a4d6ad7748cc23bab5: Status 404 returned error can't find the container with id 8c14318ba753c7f274be6e567ff7991e6eb98b9ac4f4b4a4d6ad7748cc23bab5
Oct 30 11:46:19.958591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.958563    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"]
Oct 30 11:46:19.960113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.960082    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:46:19.960194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.960144    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:46:19.960194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:19.960173    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:46:20.378973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:20.378877    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:20.533899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:20.533857    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:20.638280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:20.638192    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" event={"ID":"a9a37abf-a6cc-4e03-8490-e1c9da962c87","Type":"ContainerStarted","Data":"d9c9e521dde33be9355f6b718005b26d67b439a75e6734d391ab008d603866c0"}
Oct 30 11:46:20.638280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:20.638250    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" event={"ID":"a9a37abf-a6cc-4e03-8490-e1c9da962c87","Type":"ContainerStarted","Data":"8c14318ba753c7f274be6e567ff7991e6eb98b9ac4f4b4a4d6ad7748cc23bab5"}
Oct 30 11:46:20.700225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:20.700155    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podStartSLOduration=1.7001348090000001 podStartE2EDuration="1.700134809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:20.696971366 +0000 UTC m=+984.491855861" watchObservedRunningTime="2026-10-30 11:46:20.700134809 +0000 UTC m=+984.495019304"
Oct 30 11:46:21.638515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:21.638485    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:21.640423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:21.640398    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:46:21.640586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:21.640441    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 11:46:21.943100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:21.942996    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:22.639394 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.639361    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:46:22.639817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.639415    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 11:46:22.838559 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.838521    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"]
Oct 30 11:46:22.839210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.839192    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:22.839265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.839213    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:22.839372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.839361    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7540ff-7710-4e58-9ac0-138720256e40" containerName="cleanup"
Oct 30 11:46:22.839407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.839377    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ad2222-ce50-45ed-a431-e4e269033045" containerName="cleanup"
Oct 30 11:46:22.839407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.839389    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ce31d9-5fb5-40aa-9a1f-e719f7774a39" containerName="cleanup"
Oct 30 11:46:22.843290 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.843265    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.866627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.866569    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"openshift-service-ca.crt\""
Oct 30 11:46:22.867433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.867402    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.867597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.867490    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.867597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.867557    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hjc\" (UniqueName: \"kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.867726 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.867714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.867782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.867747    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.867836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.867789    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.871987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.871957    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"]
Oct 30 11:46:22.902535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.888214    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"kube-root-ca.crt\""
Oct 30 11:46:22.902535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.889135    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"distinct-w1-e43a66-kserve-self-signed-certs\""
Oct 30 11:46:22.902535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.889274    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"default-dockercfg-qr7jz\""
Oct 30 11:46:22.966692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.966654    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:22.968952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.968933    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hjc\" (UniqueName: \"kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969036    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969057    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969079    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969124    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969157    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969597    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969640    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.969782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.969673    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.971651 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.971630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:22.971881 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:22.971865    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:23.000204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.000141    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hjc\" (UniqueName: \"kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc\") pod \"distinct-w1-e43a66-kserve-575d766f7-wltgq\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:23.171868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.171725    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:23.568403 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:46:23.568377    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode32f971d_ffd4_4f01_b040_a6ff204125c5.slice/crio-88541a75d811abb2311d596f99a093b83bae60328f730237bd56cecc967d9eac WatchSource:0}: Error finding container 88541a75d811abb2311d596f99a093b83bae60328f730237bd56cecc967d9eac: Status 404 returned error can't find the container with id 88541a75d811abb2311d596f99a093b83bae60328f730237bd56cecc967d9eac
Oct 30 11:46:23.569418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.569397    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"]
Oct 30 11:46:23.638816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.638786    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:46:23.638950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.638848    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 11:46:23.659490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.659442    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" event={"ID":"e32f971d-ffd4-4f01-b040-a6ff204125c5","Type":"ContainerStarted","Data":"88541a75d811abb2311d596f99a093b83bae60328f730237bd56cecc967d9eac"}
Oct 30 11:46:23.871287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:23.871191    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:24.638867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:24.638831    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:46:24.639063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:24.638892    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 11:46:24.667811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:24.667719    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" event={"ID":"e32f971d-ffd4-4f01-b040-a6ff204125c5","Type":"ContainerStarted","Data":"693cb18ce1d04e679209ac7a5b7eba2fc069baea68d8e165a49fd1bc605647db"}
Oct 30 11:46:24.717364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:24.717298    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podStartSLOduration=2.717278306 podStartE2EDuration="2.717278306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:24.714607361 +0000 UTC m=+988.509491856" watchObservedRunningTime="2026-10-30 11:46:24.717278306 +0000 UTC m=+988.512162803"
Oct 30 11:46:25.173146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:25.173113    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:25.639275 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:25.639241    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:46:25.639509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:25.639301    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 11:46:25.659996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:25.659964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:26.639246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:26.639212    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:46:26.639697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:26.639269    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 11:46:27.456335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:27.456302    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:28.155352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:28.155323    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:28.155854 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:28.155831    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:28.156976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:28.156938    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:46:29.159599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:29.159564    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:30.859445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:30.859398    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:32.766819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:32.766783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:33.173017 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:33.172934    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:33.185082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:33.185057    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:33.759871 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:33.759843    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:46:34.474233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:34.474197    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:36.314812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:36.314783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:38.293370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:38.293333    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:38.731638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:38.731600    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:38.921191 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:38.921161    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"]
Oct 30 11:46:38.925306 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:38.925285    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:38.946818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:38.946790    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"e2e-conflict-b-a76a6963-kserve-self-signed-certs\""
Oct 30 11:46:38.948566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:38.948543    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"]
Oct 30 11:46:39.036094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.036010    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:39.082174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.082134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.082387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.082206    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.082493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.082381    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2pg\" (UniqueName: \"kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.082493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.082433    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.082617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.082552    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.082671 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.082630    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.183777 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.183740    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.183992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.183797    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.183992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.183837    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.183992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.183892    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.183992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.183945    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.184202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.184027    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2pg\" (UniqueName: \"kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.184261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.184200    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.184600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.184580    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.184852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.184822    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.186850 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.186824    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.187110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.187091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.209334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.209296    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2pg\" (UniqueName: \"kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg\") pod \"e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.242261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.242225    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:39.554947 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:46:39.554915    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5546ebac_c44f_41c5_b47b_b22310962519.slice/crio-e52ee2ad73e4e8145fa2e219ef4694d315eb41c2bb5dece9b78acd21bd74ddf4 WatchSource:0}: Error finding container e52ee2ad73e4e8145fa2e219ef4694d315eb41c2bb5dece9b78acd21bd74ddf4: Status 404 returned error can't find the container with id e52ee2ad73e4e8145fa2e219ef4694d315eb41c2bb5dece9b78acd21bd74ddf4
Oct 30 11:46:39.555386 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.555302    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"]
Oct 30 11:46:39.724270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.724226    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:39.791169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.791129    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8" event={"ID":"5546ebac-c44f-41c5-b47b-b22310962519","Type":"ContainerStarted","Data":"79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9"}
Oct 30 11:46:39.791169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.791180    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8" event={"ID":"5546ebac-c44f-41c5-b47b-b22310962519","Type":"ContainerStarted","Data":"e52ee2ad73e4e8145fa2e219ef4694d315eb41c2bb5dece9b78acd21bd74ddf4"}
Oct 30 11:46:39.835214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:39.835116    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8" podStartSLOduration=1.8351026240000001 podStartE2EDuration="1.835102624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:39.832527848 +0000 UTC m=+1003.627412340" watchObservedRunningTime="2026-10-30 11:46:39.835102624 +0000 UTC m=+1003.629987117"
Oct 30 11:46:40.242859 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:40.242830    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:40.397712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:40.397681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:42.170889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:42.170854    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:44.009531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:44.009491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:45.771425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:45.771392    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:47.638818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:47.638784    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:48.228308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:48.228267    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:49.242758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:49.242329    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:49.256527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:49.256501    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:49.754334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:49.754286    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:49.856601 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:46:49.856563    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:46:49.873302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:49.873274    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:46:51.560905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:51.560872    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:53.356067 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:53.356019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:54.426102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:54.426068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:55.248700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:55.248661    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:57.061309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:57.061268    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:46:58.842090 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:46:58.842045    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:00.561710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:00.561648    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:02.551126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:02.551092    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:04.468881 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:04.468847    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:06.281734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:06.281648    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:08.145995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:08.145946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:09.937394 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:09.937355    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:10.948177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.948141    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"]
Oct 30 11:47:10.953481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.953440    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:10.971003 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.970977    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"]
Oct 30 11:47:10.979040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.979019    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"distinct-w0-3ccdab-kserve-self-signed-certs\""
Oct 30 11:47:10.994592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.994563    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xwz\" (UniqueName: \"kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:10.994710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.994608    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:10.994710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.994705    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:10.994829 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.994807    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:10.994868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.994848    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:10.994953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:10.994939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.033320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.033284    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:11.096009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.095970    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096032    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096065    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096103    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096241    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xwz\" (UniqueName: \"kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096274    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096514 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096488    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096766    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.096932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.096851    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.099029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.099005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.099140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.099062    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.122384 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.122358    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xwz\" (UniqueName: \"kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz\") pod \"distinct-w0-3ccdab-kserve-5f67484c47-bnchp\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.269380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.269294    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:11.608680 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:47:11.608649    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode52f4f19_a1b7_4948_b692_8560d0d8b846.slice/crio-bbfe1a08dcaef4a8c2f63e6d3fe4bb5b48559e40c8739c0cb7ad96d66b4a13e7 WatchSource:0}: Error finding container bbfe1a08dcaef4a8c2f63e6d3fe4bb5b48559e40c8739c0cb7ad96d66b4a13e7: Status 404 returned error can't find the container with id bbfe1a08dcaef4a8c2f63e6d3fe4bb5b48559e40c8739c0cb7ad96d66b4a13e7
Oct 30 11:47:11.609121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.609098    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"]
Oct 30 11:47:11.846897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:11.846859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:12.036076 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:12.036036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" event={"ID":"e52f4f19-a1b7-4948-b692-8560d0d8b846","Type":"ContainerStarted","Data":"87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32"}
Oct 30 11:47:12.036503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:12.036084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" event={"ID":"e52f4f19-a1b7-4948-b692-8560d0d8b846","Type":"ContainerStarted","Data":"bbfe1a08dcaef4a8c2f63e6d3fe4bb5b48559e40c8739c0cb7ad96d66b4a13e7"}
Oct 30 11:47:12.078886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:12.078839    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podStartSLOduration=2.078825868 podStartE2EDuration="2.078825868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:12.077819079 +0000 UTC m=+1035.872703574" watchObservedRunningTime="2026-10-30 11:47:12.078825868 +0000 UTC m=+1035.873710378"
Oct 30 11:47:12.270154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:12.270116    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:13.158607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:13.158576    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"]
Oct 30 11:47:13.238359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:13.237995    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:13.777057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:13.777028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:15.379640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.379605    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"]
Oct 30 11:47:15.384026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.384003    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.399296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.399266    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-849669-openshift-default-dockercfg-2qnx2\""
Oct 30 11:47:15.417086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.417049    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"]
Oct 30 11:47:15.444868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.444828    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445051 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.444898    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445042    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445164    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445194    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445283    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sgf\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445328    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.445483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.445425    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546508 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546539    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546660    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546917 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sgf\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546917 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546849    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.546917 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.547102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.546949    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.547102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.547003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.547102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.547042    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.547277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.547189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.547502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.547465    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.547745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.547725    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.548936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.548913    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.549424 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.549400    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.571945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.571915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.572089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.571953    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sgf\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:15.657711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.657622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:15.700552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:15.700520    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:16.027354 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:47:16.027311    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f70fc3_88ee_4ec1_9597_49e6413a827b.slice/crio-4667fb4d6f62c434565e2d9e8dc7c80b0e5fdc86be49c6c0a31f711524a776f6 WatchSource:0}: Error finding container 4667fb4d6f62c434565e2d9e8dc7c80b0e5fdc86be49c6c0a31f711524a776f6: Status 404 returned error can't find the container with id 4667fb4d6f62c434565e2d9e8dc7c80b0e5fdc86be49c6c0a31f711524a776f6
Oct 30 11:47:16.027744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:16.027722    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"]
Oct 30 11:47:16.030258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:16.030222    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:16.030334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:16.030303    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:16.030370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:16.030341    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:16.063487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:16.063446    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" event={"ID":"55f70fc3-88ee-4ec1-9597-49e6413a827b","Type":"ContainerStarted","Data":"4667fb4d6f62c434565e2d9e8dc7c80b0e5fdc86be49c6c0a31f711524a776f6"}
Oct 30 11:47:17.072270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:17.072233    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" event={"ID":"55f70fc3-88ee-4ec1-9597-49e6413a827b","Type":"ContainerStarted","Data":"e48a28c0d746134ed09b8413666cad52530553c1f7abffcdc29ffe68b5937365"}
Oct 30 11:47:17.462779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:17.462741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:17.702476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:17.701840    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:17.708787 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:17.708744    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:47:17.708991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:17.708810    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 11:47:18.702601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:18.702563    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:47:18.703097 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:18.702627    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 11:47:19.514476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.506268    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:19.529182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.529119    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podStartSLOduration=4.529097626 podStartE2EDuration="4.529097626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:17.167962937 +0000 UTC m=+1040.962847433" watchObservedRunningTime="2026-10-30 11:47:19.529097626 +0000 UTC m=+1043.323982122"
Oct 30 11:47:19.533093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.533044    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"]
Oct 30 11:47:19.538500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.538471    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.568017 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.567248    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"]
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.640865    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.640918    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.640943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpmk\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.640973    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.641009    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.641034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.641183    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.641257    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.641285    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.703477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.701146    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:47:19.703477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.701205    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 11:47:19.742574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742537    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742612    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742635    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742655    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742671    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpmk\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742690    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.742751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.742730    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.743072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.743057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.743544 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.743529    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.744241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.743741    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.744241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.743916    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.744241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.744183    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.749472 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.747530    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.749472 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.747679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.770657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.770521    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.772163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.772137    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpmk\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:19.799475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.798982    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:19.864162 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:47:19.864088    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:47:19.869655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.869426    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:20.005148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:19.999653    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"]
Oct 30 11:47:20.093507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.087069    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"]
Oct 30 11:47:20.093507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.087727    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.093507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.087877    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnf5r\" (UniqueName: \"kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.093507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.087953    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.093507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.088111    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.127703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.123960    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:47:20.127703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.124028    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:47:20.127703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.124115    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:47:20.127703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.124836    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:47:20.205915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.205421    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.205915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.205623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnf5r\" (UniqueName: \"kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.205915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.205708    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.263233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.253510    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.263233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.263150    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnf5r\" (UniqueName: \"kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.275635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.275542    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e38c024-k7qgk\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.397728 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.360962    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:20.677480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.662415    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"]
Oct 30 11:47:20.725899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.725739    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:47:20.726343 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.726137    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 11:47:20.744244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.744188    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:20.744600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.744566    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:20.756525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:20.756480    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:21.273711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.272871    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:21.421646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.421610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:21.558234 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.558151    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:21.585404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.581596    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"]
Oct 30 11:47:21.691478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.672505    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" event={"ID":"dcc40b82-8881-400c-813e-1f01248c02f1","Type":"ContainerStarted","Data":"3d3de549e7c7bacaae76e675f46a24acefefefb074c8014076b55b6b9a000398"}
Oct 30 11:47:21.691478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.672575    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" event={"ID":"dcc40b82-8881-400c-813e-1f01248c02f1","Type":"ContainerStarted","Data":"e137f83f1a5ba79045d5d08b51a7fbac89697cefcfabd7c749caba1748356611"}
Oct 30 11:47:21.704102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.704072    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:47:21.704295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.704269    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 11:47:21.749948 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.749920    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:47:21.756837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.755060    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podStartSLOduration=2.755039721 podStartE2EDuration="2.755039721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:21.735021686 +0000 UTC m=+1045.529906188" watchObservedRunningTime="2026-10-30 11:47:21.755039721 +0000 UTC m=+1045.549924228"
Oct 30 11:47:21.870582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:21.870502    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:22.165210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.165119    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:22.683604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.683572    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk" event={"ID":"36cb5328-fab9-41f4-90e6-cd185ae5feaa","Type":"ContainerStarted","Data":"970b9f1fae18716c6967028d02a4bf82bbfa0f992f35d0924139fe370319fe05"}
Oct 30 11:47:22.683906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.683878    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk" event={"ID":"36cb5328-fab9-41f4-90e6-cd185ae5feaa","Type":"ContainerStarted","Data":"39a29f68cb543c0f8a282613f2f2b8de00d6ec7cd320b4865bbca970b84bcbbf"}
Oct 30 11:47:22.708035 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.708004    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:47:22.708803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.708292    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 11:47:22.728652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.728585    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk" podStartSLOduration=3.728571009 podStartE2EDuration="3.728571009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:22.725654006 +0000 UTC m=+1046.520538513" watchObservedRunningTime="2026-10-30 11:47:22.728571009 +0000 UTC m=+1046.523455502"
Oct 30 11:47:22.871645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.871280    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:22.871645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:22.871342    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:23.696282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:23.694693    5060 generic.go:356] "Generic (PLEG): container finished" podID="36cb5328-fab9-41f4-90e6-cd185ae5feaa" containerID="970b9f1fae18716c6967028d02a4bf82bbfa0f992f35d0924139fe370319fe05" exitCode=0
Oct 30 11:47:23.696282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:23.696054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk" event={"ID":"36cb5328-fab9-41f4-90e6-cd185ae5feaa","Type":"ContainerDied","Data":"970b9f1fae18716c6967028d02a4bf82bbfa0f992f35d0924139fe370319fe05"}
Oct 30 11:47:23.870389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:23.870348    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:23.870618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:23.870485    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:24.120534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.119700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:24.217478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.216652    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"]
Oct 30 11:47:24.229628 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.227809    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.244670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.244408    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"]
Oct 30 11:47:24.298475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.297788    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.299167    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.299892    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.299937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzwd\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.299982    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.300012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.300052    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.300080    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.300112    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.300148    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.300189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.319982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.317303    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:47:24.415527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.408751    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.415799 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.415773    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzwd\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.416910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.411330    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.417072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.416477    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.417240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.417221    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.417731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.417714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.417874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.417860    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.417989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.417976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.418135 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.418122    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.419180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.419154    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.419283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.416782    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.419283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.419104    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.419283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.417643    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.419703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.419669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.426830 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.426773    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.431106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.431080    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.444063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.443247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzwd\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.446057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.445980    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.553347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.553309    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:24.870729 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.870695    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:24.870936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.870755    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:24.919950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.919522    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"]
Oct 30 11:47:24.921562 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:47:24.921520    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672e173a_4991_4411_a7cf_306fbad68ccd.slice/crio-a767685976b4f719e7fc7eaf1f5a13e8529d66d8c09d8a757cba034f6a42e70e WatchSource:0}: Error finding container a767685976b4f719e7fc7eaf1f5a13e8529d66d8c09d8a757cba034f6a42e70e: Status 404 returned error can't find the container with id a767685976b4f719e7fc7eaf1f5a13e8529d66d8c09d8a757cba034f6a42e70e
Oct 30 11:47:24.924289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.924256    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:24.924401 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.924337    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:24.924487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.924440    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:24.932350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:24.932324    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:25.030740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.030711    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca\") pod \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") "
Oct 30 11:47:25.030870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.030754    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca\") pod \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") "
Oct 30 11:47:25.030870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.030827    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r\" (UniqueName: \"kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r\") pod \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\" (UID: \"36cb5328-fab9-41f4-90e6-cd185ae5feaa\") "
Oct 30 11:47:25.031105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.031077    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca" pod "36cb5328-fab9-41f4-90e6-cd185ae5feaa" (UID: "36cb5328-fab9-41f4-90e6-cd185ae5feaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:47:25.031419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.031367    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:25.033551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.033530    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca" pod "36cb5328-fab9-41f4-90e6-cd185ae5feaa" (UID: "36cb5328-fab9-41f4-90e6-cd185ae5feaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:47:25.033732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.033705    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r" pod "36cb5328-fab9-41f4-90e6-cd185ae5feaa" (UID: "36cb5328-fab9-41f4-90e6-cd185ae5feaa"). InnerVolumeSpecName "kube-api-access-jnf5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:47:25.132822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.132787    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cb5328-fab9-41f4-90e6-cd185ae5feaa-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:25.132822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.132817    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnf5r\" (UniqueName: \"kubernetes.io/projected/36cb5328-fab9-41f4-90e6-cd185ae5feaa-kube-api-access-jnf5r\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:25.728382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.728347    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"
Oct 30 11:47:25.728382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.728368    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk" event={"ID":"36cb5328-fab9-41f4-90e6-cd185ae5feaa","Type":"ContainerDied","Data":"39a29f68cb543c0f8a282613f2f2b8de00d6ec7cd320b4865bbca970b84bcbbf"}
Oct 30 11:47:25.728647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.728437    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a29f68cb543c0f8a282613f2f2b8de00d6ec7cd320b4865bbca970b84bcbbf"
Oct 30 11:47:25.730697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.730661    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" event={"ID":"672e173a-4991-4411-a7cf-306fbad68ccd","Type":"ContainerStarted","Data":"5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a"}
Oct 30 11:47:25.730862 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.730726    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" event={"ID":"672e173a-4991-4411-a7cf-306fbad68ccd","Type":"ContainerStarted","Data":"a767685976b4f719e7fc7eaf1f5a13e8529d66d8c09d8a757cba034f6a42e70e"}
Oct 30 11:47:25.813918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.813857    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podStartSLOduration=1.813839515 podStartE2EDuration="1.813839515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:25.798940635 +0000 UTC m=+1049.593825130" watchObservedRunningTime="2026-10-30 11:47:25.813839515 +0000 UTC m=+1049.608724008"
Oct 30 11:47:25.875882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.875847    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:25.876101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:25.875903    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:26.202539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:26.198511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:26.554888 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:26.554562    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:26.555852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:26.555830    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 11:47:26.556027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:26.556005    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 11:47:26.871536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:26.870575    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:26.871536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:26.870628    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:27.554570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:27.554531    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 11:47:27.555067 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:27.554587    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 11:47:27.869795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:27.869716    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:27.869795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:27.869777    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:28.187917 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.187819    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8" podUID="5546ebac-c44f-41c5-b47b-b22310962519" containerName="main" containerID="cri-o://79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9" gracePeriod=45
Oct 30 11:47:28.331672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.331587    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:28.554604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.554571    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 11:47:28.554942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.554622    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 11:47:28.572119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.572085    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:47:28.623332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.622196    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs\" (UniqueName: \"kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs\") pod \"5546ebac-c44f-41c5-b47b-b22310962519\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") "
Oct 30 11:47:28.623332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.622311    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home\") pod \"5546ebac-c44f-41c5-b47b-b22310962519\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") "
Oct 30 11:47:28.623332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.622354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache\") pod \"5546ebac-c44f-41c5-b47b-b22310962519\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") "
Oct 30 11:47:28.623332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.622402    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm\") pod \"5546ebac-c44f-41c5-b47b-b22310962519\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") "
Oct 30 11:47:28.623332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.622493    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg\" (UniqueName: \"kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg\") pod \"5546ebac-c44f-41c5-b47b-b22310962519\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") "
Oct 30 11:47:28.623332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.622539    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir\") pod \"5546ebac-c44f-41c5-b47b-b22310962519\" (UID: \"5546ebac-c44f-41c5-b47b-b22310962519\") "
Oct 30 11:47:28.623789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.623598    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home" pod "5546ebac-c44f-41c5-b47b-b22310962519" (UID: "5546ebac-c44f-41c5-b47b-b22310962519"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:47:28.624423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.624318    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache" pod "5546ebac-c44f-41c5-b47b-b22310962519" (UID: "5546ebac-c44f-41c5-b47b-b22310962519"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:47:28.624657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.624634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir" pod "5546ebac-c44f-41c5-b47b-b22310962519" (UID: "5546ebac-c44f-41c5-b47b-b22310962519"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:47:28.626982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.626961    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm" pod "5546ebac-c44f-41c5-b47b-b22310962519" (UID: "5546ebac-c44f-41c5-b47b-b22310962519"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:47:28.627858 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.627835    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg" pod "5546ebac-c44f-41c5-b47b-b22310962519" (UID: "5546ebac-c44f-41c5-b47b-b22310962519"). InnerVolumeSpecName "kube-api-access-dm2pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:47:28.628202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.628179    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs" pod "5546ebac-c44f-41c5-b47b-b22310962519" (UID: "5546ebac-c44f-41c5-b47b-b22310962519"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:47:28.731497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.731314    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:28.731497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.731343    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:28.731497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.731354    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm2pg\" (UniqueName: \"kubernetes.io/projected/5546ebac-c44f-41c5-b47b-b22310962519-kube-api-access-dm2pg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:28.731497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.731363    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:28.731497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.731372    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5546ebac-c44f-41c5-b47b-b22310962519-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:28.731497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.731379    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5546ebac-c44f-41c5-b47b-b22310962519-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:28.761631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.761587    5060 generic.go:356] "Generic (PLEG): container finished" podID="5546ebac-c44f-41c5-b47b-b22310962519" containerID="79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9" exitCode=0
Oct 30 11:47:28.761817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.761742    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8" event={"ID":"5546ebac-c44f-41c5-b47b-b22310962519","Type":"ContainerDied","Data":"79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9"}
Oct 30 11:47:28.761817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.761770    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8" event={"ID":"5546ebac-c44f-41c5-b47b-b22310962519","Type":"ContainerDied","Data":"e52ee2ad73e4e8145fa2e219ef4694d315eb41c2bb5dece9b78acd21bd74ddf4"}
Oct 30 11:47:28.761817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.761786    5060 scope.go:122] "RemoveContainer" containerID="79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9"
Oct 30 11:47:28.761993 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.761891    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"
Oct 30 11:47:28.800697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.798601    5060 scope.go:122] "RemoveContainer" containerID="79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9"
Oct 30 11:47:28.800697 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:47:28.798994    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9\": container with ID starting with 79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9 not found: ID does not exist" containerID="79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9"
Oct 30 11:47:28.800697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.799032    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9"} err="failed to get container status \"79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9\": rpc error: code = NotFound desc = could not find container \"79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9\": container with ID starting with 79e1d4dc3fade88b7734426b8b41faeb5c4703229e68dc29cb78dd36870674b9 not found: ID does not exist"
Oct 30 11:47:28.827880 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.824900    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"]
Oct 30 11:47:28.827880 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.827783    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-b-a76a6963-kserve-5846b9cfd4-gbxj8"]
Oct 30 11:47:28.870803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.870773    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 11:47:28.870999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:28.870827    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 11:47:29.554131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:29.554088    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 11:47:29.554292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:29.554146    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 11:47:29.874654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:29.874576    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:29.875015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:29.874874    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:29.875531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:29.875512    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 11:47:30.135295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.135191    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:30.336985 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.336957    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5546ebac-c44f-41c5-b47b-b22310962519" path="/var/lib/kubelet/pods/5546ebac-c44f-41c5-b47b-b22310962519/volumes"
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.403506    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"]
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.404241    5060 cpu_manager.go:414] "containerMap: removing container" podUID="36cb5328-fab9-41f4-90e6-cd185ae5feaa" containerName="revoke-keys"
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.404261    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cb5328-fab9-41f4-90e6-cd185ae5feaa" containerName="revoke-keys"
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.404280    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5546ebac-c44f-41c5-b47b-b22310962519" containerName="main"
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.404288    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5546ebac-c44f-41c5-b47b-b22310962519" containerName="main"
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.404475    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cb5328-fab9-41f4-90e6-cd185ae5feaa" containerName="revoke-keys"
Oct 30 11:47:30.407523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.404488    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5546ebac-c44f-41c5-b47b-b22310962519" containerName="main"
Oct 30 11:47:30.417479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.413757    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.419701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.419666    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"]
Oct 30 11:47:30.433358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.433325    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:47:30.433648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.433623    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:47:30.434565 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.434545    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:47:30.434935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.434911    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:47:30.450943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.450700    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.450943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.450748    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4sx\" (UniqueName: \"kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.450943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.450788    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.552220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.552184    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.552412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.552238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4sx\" (UniqueName: \"kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.552412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.552278    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.553010 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.552985    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.554113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.554091    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 11:47:30.554242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.554130    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 11:47:30.555159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.555137    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.576674    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4sx\" (UniqueName: \"kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx\") pod \"maas-api-revoke-sub-0ca34736-q8x78\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:30.737725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:30.737689    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:31.068073 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:47:31.068040    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff4ae3f_4f4d_41ce_9356_ea273a700b84.slice/crio-a33327a5d4f50dac2add9d4bd00eecf0ff9284fede2542d8a4621597c9d3c2dc WatchSource:0}: Error finding container a33327a5d4f50dac2add9d4bd00eecf0ff9284fede2542d8a4621597c9d3c2dc: Status 404 returned error can't find the container with id a33327a5d4f50dac2add9d4bd00eecf0ff9284fede2542d8a4621597c9d3c2dc
Oct 30 11:47:31.068540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.068517    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"]
Oct 30 11:47:31.554287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.554252    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 11:47:31.554480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.554305    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 11:47:31.789581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.789523    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78" event={"ID":"3ff4ae3f-4f4d-41ce-9356-ea273a700b84","Type":"ContainerStarted","Data":"785f98f83947cc03176f6f0a40eff5e59e139e30c9f44a5940d57737c3161345"}
Oct 30 11:47:31.789768 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.789666    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78" event={"ID":"3ff4ae3f-4f4d-41ce-9356-ea273a700b84","Type":"ContainerStarted","Data":"a33327a5d4f50dac2add9d4bd00eecf0ff9284fede2542d8a4621597c9d3c2dc"}
Oct 30 11:47:31.856123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.856068    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78" podStartSLOduration=1.856043594 podStartE2EDuration="1.856043594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:31.853647431 +0000 UTC m=+1055.648531925" watchObservedRunningTime="2026-10-30 11:47:31.856043594 +0000 UTC m=+1055.650928143"
Oct 30 11:47:31.908697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:31.908659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:32.070552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:32.070435    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:32.729474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:32.728816    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:32.730502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:32.730429    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:32.731232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:32.731209    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:47:32.797826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:32.797792    5060 generic.go:356] "Generic (PLEG): container finished" podID="3ff4ae3f-4f4d-41ce-9356-ea273a700b84" containerID="785f98f83947cc03176f6f0a40eff5e59e139e30c9f44a5940d57737c3161345" exitCode=0
Oct 30 11:47:32.797976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:32.797882    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78" event={"ID":"3ff4ae3f-4f4d-41ce-9356-ea273a700b84","Type":"ContainerDied","Data":"785f98f83947cc03176f6f0a40eff5e59e139e30c9f44a5940d57737c3161345"}
Oct 30 11:47:33.951720 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.951694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:33.981223 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.981195    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:33.993268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.993228    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx\" (UniqueName: \"kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx\") pod \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") "
Oct 30 11:47:33.993418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.993377    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca\") pod \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") "
Oct 30 11:47:33.993545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.993530    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca\") pod \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\" (UID: \"3ff4ae3f-4f4d-41ce-9356-ea273a700b84\") "
Oct 30 11:47:33.994255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.994187    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca" pod "3ff4ae3f-4f4d-41ce-9356-ea273a700b84" (UID: "3ff4ae3f-4f4d-41ce-9356-ea273a700b84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:47:33.996077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.996043    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx" pod "3ff4ae3f-4f4d-41ce-9356-ea273a700b84" (UID: "3ff4ae3f-4f4d-41ce-9356-ea273a700b84"). InnerVolumeSpecName "kube-api-access-sw4sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:47:33.996077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:33.996056    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca" pod "3ff4ae3f-4f4d-41ce-9356-ea273a700b84" (UID: "3ff4ae3f-4f4d-41ce-9356-ea273a700b84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:47:34.095476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:34.095371    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:34.095476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:34.095412    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:34.095476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:34.095428    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw4sx\" (UniqueName: \"kubernetes.io/projected/3ff4ae3f-4f4d-41ce-9356-ea273a700b84-kube-api-access-sw4sx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:34.816941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:34.816909    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"
Oct 30 11:47:34.817126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:34.816903    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78" event={"ID":"3ff4ae3f-4f4d-41ce-9356-ea273a700b84","Type":"ContainerDied","Data":"a33327a5d4f50dac2add9d4bd00eecf0ff9284fede2542d8a4621597c9d3c2dc"}
Oct 30 11:47:34.817126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:34.817041    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a33327a5d4f50dac2add9d4bd00eecf0ff9284fede2542d8a4621597c9d3c2dc"
Oct 30 11:47:35.754518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:35.754481    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:37.471063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:37.471018    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:39.716628 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:39.716565    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:40.152542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:40.151742    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:40.254603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:40.254556    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:40.361642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:40.361592    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:40.447036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:40.446996    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:40.899214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:40.899047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:41.533009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:41.532972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:42.760772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:42.760730    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:44.287498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.287444    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:44.746182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.746138    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"]
Oct 30 11:47:44.746747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.746727    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3ff4ae3f-4f4d-41ce-9356-ea273a700b84" containerName="revoke-keys"
Oct 30 11:47:44.746747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.746747    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff4ae3f-4f4d-41ce-9356-ea273a700b84" containerName="revoke-keys"
Oct 30 11:47:44.746931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.746918    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff4ae3f-4f4d-41ce-9356-ea273a700b84" containerName="revoke-keys"
Oct 30 11:47:44.750539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.750517    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.786756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.786078    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"]
Oct 30 11:47:44.805781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.805752    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:47:44.806048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.805766    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:47:44.806124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.805768    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:47:44.806124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.805833    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:47:44.807171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.807124    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:44.864055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.864021    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.864250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.864067    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.864250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.864087    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97m7\" (UniqueName: \"kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.965471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.965408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.965624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.965512    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.965624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.965543    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97m7\" (UniqueName: \"kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.966274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.966249    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.967985 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.967961    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:44.994438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:44.994410    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97m7\" (UniqueName: \"kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7\") pod \"maas-api-revoke-sub-ad352c37-wczrr\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:45.076205 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:45.076112    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:45.382046 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:47:45.382014    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63acc2a_1b05_45b5_b364_1c9972732ddc.slice/crio-8a40832cf4b5e53347db603a06eea8560414727438067a70fb3259807de87da7 WatchSource:0}: Error finding container 8a40832cf4b5e53347db603a06eea8560414727438067a70fb3259807de87da7: Status 404 returned error can't find the container with id 8a40832cf4b5e53347db603a06eea8560414727438067a70fb3259807de87da7
Oct 30 11:47:45.382495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:45.382231    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"]
Oct 30 11:47:45.987937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:45.987890    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr" event={"ID":"c63acc2a-1b05-45b5-b364-1c9972732ddc","Type":"ContainerStarted","Data":"4ba2da578c2c5533e269ad11ee48f2c0fb253d9e9e1a21b86920dfb0bf45e79b"}
Oct 30 11:47:45.988124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:45.987996    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr" event={"ID":"c63acc2a-1b05-45b5-b364-1c9972732ddc","Type":"ContainerStarted","Data":"8a40832cf4b5e53347db603a06eea8560414727438067a70fb3259807de87da7"}
Oct 30 11:47:46.030426 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:46.030356    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr" podStartSLOduration=2.030340903 podStartE2EDuration="2.030340903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:46.029748203 +0000 UTC m=+1069.824632698" watchObservedRunningTime="2026-10-30 11:47:46.030340903 +0000 UTC m=+1069.825225398"
Oct 30 11:47:46.647131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:46.647028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:46.996723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:46.996687    5060 generic.go:356] "Generic (PLEG): container finished" podID="c63acc2a-1b05-45b5-b364-1c9972732ddc" containerID="4ba2da578c2c5533e269ad11ee48f2c0fb253d9e9e1a21b86920dfb0bf45e79b" exitCode=0
Oct 30 11:47:46.996909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:46.996766    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr" event={"ID":"c63acc2a-1b05-45b5-b364-1c9972732ddc","Type":"ContainerDied","Data":"4ba2da578c2c5533e269ad11ee48f2c0fb253d9e9e1a21b86920dfb0bf45e79b"}
Oct 30 11:47:47.695643 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.695607    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"]
Oct 30 11:47:47.699645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.699597    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.729689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.729657    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"]
Oct 30 11:47:47.798379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.798350    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ccdab-openshift-default-dockercfg-zcvfb\""
Oct 30 11:47:47.813044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813015    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813075    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813163    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813264    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjlw\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813348 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813403 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813388    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813467 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.813552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.813499    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914646    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914664    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjlw\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914782    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914851    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.914893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914876    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914927    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.914980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.915106    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.915372    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.915389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.915429    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.915603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.915594    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.917698 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.917670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.917935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.917916    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.939313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.939267    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:47.939521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:47.939389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjlw\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:48.027927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.027891    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:48.169695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.169664    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:48.321742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.321651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca\") pod \"c63acc2a-1b05-45b5-b364-1c9972732ddc\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") "
Oct 30 11:47:48.321920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.321758    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7\" (UniqueName: \"kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7\") pod \"c63acc2a-1b05-45b5-b364-1c9972732ddc\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") "
Oct 30 11:47:48.322651 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.322101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca" pod "c63acc2a-1b05-45b5-b364-1c9972732ddc" (UID: "c63acc2a-1b05-45b5-b364-1c9972732ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:47:48.322651 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.322330    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca\") pod \"c63acc2a-1b05-45b5-b364-1c9972732ddc\" (UID: \"c63acc2a-1b05-45b5-b364-1c9972732ddc\") "
Oct 30 11:47:48.323944 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.323915    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:48.336415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.336374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca" pod "c63acc2a-1b05-45b5-b364-1c9972732ddc" (UID: "c63acc2a-1b05-45b5-b364-1c9972732ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:47:48.339105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.339074    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7" pod "c63acc2a-1b05-45b5-b364-1c9972732ddc" (UID: "c63acc2a-1b05-45b5-b364-1c9972732ddc"). InnerVolumeSpecName "kube-api-access-d97m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:47:48.355640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.355605    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:48.368041 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:47:48.368013    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1ac9d1_de57_4606_aacb_aaf42086c064.slice/crio-9518420e2db2b85cec5c9cf85817ad451ced82dce9ff65510333a28b5b30808f WatchSource:0}: Error finding container 9518420e2db2b85cec5c9cf85817ad451ced82dce9ff65510333a28b5b30808f: Status 404 returned error can't find the container with id 9518420e2db2b85cec5c9cf85817ad451ced82dce9ff65510333a28b5b30808f
Oct 30 11:47:48.369110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.369080    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"]
Oct 30 11:47:48.372073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.372040    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:48.372176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.372104    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:48.372176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.372147    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:47:48.424816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.424786    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63acc2a-1b05-45b5-b364-1c9972732ddc-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:48.424816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.424814    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97m7\" (UniqueName: \"kubernetes.io/projected/c63acc2a-1b05-45b5-b364-1c9972732ddc-kube-api-access-d97m7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:47:48.491988 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:48.491938    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:49.012263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.012229    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr" event={"ID":"c63acc2a-1b05-45b5-b364-1c9972732ddc","Type":"ContainerDied","Data":"8a40832cf4b5e53347db603a06eea8560414727438067a70fb3259807de87da7"}
Oct 30 11:47:49.012822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.012292    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a40832cf4b5e53347db603a06eea8560414727438067a70fb3259807de87da7"
Oct 30 11:47:49.012822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.012243    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"
Oct 30 11:47:49.015182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.015143    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" event={"ID":"6a1ac9d1-de57-4606-aacb-aaf42086c064","Type":"ContainerStarted","Data":"d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d"}
Oct 30 11:47:49.015317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.015199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" event={"ID":"6a1ac9d1-de57-4606-aacb-aaf42086c064","Type":"ContainerStarted","Data":"9518420e2db2b85cec5c9cf85817ad451ced82dce9ff65510333a28b5b30808f"}
Oct 30 11:47:49.028750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.028715    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:49.030633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.030607    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:49.030786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.030664    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:49.088492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:49.085584    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podStartSLOduration=2.085563825 podStartE2EDuration="2.085563825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:49.076624558 +0000 UTC m=+1072.871509052" watchObservedRunningTime="2026-10-30 11:47:49.085563825 +0000 UTC m=+1072.880448323"
Oct 30 11:47:49.856959 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:47:49.856924    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:47:50.028772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:50.028740    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:50.029177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:50.028792    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:50.400480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:50.393106    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:51.029063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:51.029030    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:51.029699 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:51.029663    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:52.028641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:52.028595    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:52.028825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:52.028669    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:52.126728 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:52.126677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:53.029204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:53.029161    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:53.029391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:53.029218    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:54.028388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:54.028347    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:54.028875 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:54.028403    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:54.150489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:54.150426    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:55.029475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.028854    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 11:47:55.029475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.028919    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 11:47:55.381532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.372949    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"]
Oct 30 11:47:55.381532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.373801    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c63acc2a-1b05-45b5-b364-1c9972732ddc" containerName="revoke-keys"
Oct 30 11:47:55.381532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.373822    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63acc2a-1b05-45b5-b364-1c9972732ddc" containerName="revoke-keys"
Oct 30 11:47:55.381532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.374020    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63acc2a-1b05-45b5-b364-1c9972732ddc" containerName="revoke-keys"
Oct 30 11:47:55.404476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.398317    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.425479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.414800    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:47:55.425479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.420339    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"]
Oct 30 11:47:55.464492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.462831    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:47:55.464492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.462990    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:47:55.464492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.463061    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:47:55.627874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.627826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.628263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.628241    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.628440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.628421    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csb2c\" (UniqueName: \"kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.732782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.732747    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.756740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.756700    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.757632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.757607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csb2c\" (UniqueName: \"kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.758254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.733847    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.764063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.764033    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:55.835306 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:55.835262    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csb2c\" (UniqueName: \"kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c\") pod \"maas-api-revoke-sub-2291d5f1-9rjll\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:56.033256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.033161    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:47:56.314893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.314794    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 11:47:56.344637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.344572    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 11:47:56.470935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.469562    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:56.631621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.631557    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:56.693604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.693552    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:56.755769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.754567    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:56.755769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:56.754891    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.176573 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.174605    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.264854 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.261589    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.359646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.358642    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:47:57.359646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.358744    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.360072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.359963    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:47:57.360072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.360012    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.376611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.375140    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.376611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.375632    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:47:57.516739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.516684    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.517426 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.517403    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:47:57.789358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.789324    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:47:57.838809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.838776    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"]
Oct 30 11:47:57.997541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:57.902902    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:58.042477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.012746    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:47:58.042477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.012841    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:58.042477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.012896    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:47:58.042477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.017043    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:58.078595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.078567    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:47:58.079725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.079702    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:47:58.415211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:58.415170    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll" event={"ID":"08653d8e-8504-43da-bf07-f5688268af6e","Type":"ContainerStarted","Data":"2a49db65718730ebfd04c8f2bc07fe81d7932529ccdcf808a3987db292f8cce4"}
Oct 30 11:47:59.098808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:59.098761    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:47:59.432078 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:59.431976    5060 generic.go:356] "Generic (PLEG): container finished" podID="08653d8e-8504-43da-bf07-f5688268af6e" containerID="f4807a4dfbb8a48904da66516d5db681ea80666698368ba49bf55fb5865791e7" exitCode=0
Oct 30 11:47:59.432592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:47:59.432105    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll" event={"ID":"08653d8e-8504-43da-bf07-f5688268af6e","Type":"ContainerDied","Data":"f4807a4dfbb8a48904da66516d5db681ea80666698368ba49bf55fb5865791e7"}
Oct 30 11:48:00.618200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.618173    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:48:00.667525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.667491    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca\") pod \"08653d8e-8504-43da-bf07-f5688268af6e\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") "
Oct 30 11:48:00.667732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.667548    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c\" (UniqueName: \"kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c\") pod \"08653d8e-8504-43da-bf07-f5688268af6e\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") "
Oct 30 11:48:00.667732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.667616    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca\") pod \"08653d8e-8504-43da-bf07-f5688268af6e\" (UID: \"08653d8e-8504-43da-bf07-f5688268af6e\") "
Oct 30 11:48:00.668061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.668032    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca" pod "08653d8e-8504-43da-bf07-f5688268af6e" (UID: "08653d8e-8504-43da-bf07-f5688268af6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:48:00.670493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.670423    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca" pod "08653d8e-8504-43da-bf07-f5688268af6e" (UID: "08653d8e-8504-43da-bf07-f5688268af6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:48:00.673644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.673614    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c" pod "08653d8e-8504-43da-bf07-f5688268af6e" (UID: "08653d8e-8504-43da-bf07-f5688268af6e"). InnerVolumeSpecName "kube-api-access-csb2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:00.769571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.769531    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:00.769571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.769568    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csb2c\" (UniqueName: \"kubernetes.io/projected/08653d8e-8504-43da-bf07-f5688268af6e-kube-api-access-csb2c\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:00.769789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:00.769583    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08653d8e-8504-43da-bf07-f5688268af6e-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.127616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:01.127515    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:01.448466 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:01.448420    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"
Oct 30 11:48:01.448665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:01.448482    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll" event={"ID":"08653d8e-8504-43da-bf07-f5688268af6e","Type":"ContainerDied","Data":"2a49db65718730ebfd04c8f2bc07fe81d7932529ccdcf808a3987db292f8cce4"}
Oct 30 11:48:01.448665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:01.448536    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a49db65718730ebfd04c8f2bc07fe81d7932529ccdcf808a3987db292f8cce4"
Oct 30 11:48:02.960239 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:02.960198    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:03.195501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.195467    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"]
Oct 30 11:48:03.196422 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.196402    5060 cpu_manager.go:414] "containerMap: removing container" podUID="08653d8e-8504-43da-bf07-f5688268af6e" containerName="revoke-keys"
Oct 30 11:48:03.196505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.196427    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08653d8e-8504-43da-bf07-f5688268af6e" containerName="revoke-keys"
Oct 30 11:48:03.196656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.196643    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="08653d8e-8504-43da-bf07-f5688268af6e" containerName="revoke-keys"
Oct 30 11:48:03.200929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.200902    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.221254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.221171    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"]
Oct 30 11:48:03.303073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303033    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303073    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303307    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303472 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303418    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303513    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303666 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303648    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2q7m\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.303747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.303735    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405472    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405541    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405653    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405713    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q7m\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405819    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.405918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.405898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.406155 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.406125    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.406255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.406235    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.406422 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.406385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.406739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.406716    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.407078 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.407053    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.409272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.409247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.409577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.409555    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.430808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.430776    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.430947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.430847    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q7m\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.515235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.515132    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:03.837487 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:03.837416    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2c3a63_dd7a_43cc_96a2_7186fd13763e.slice/crio-2d37b528c25102ce628de85ce4f55e6d4fec1b9cfc69212efd0f9a539a8b856f WatchSource:0}: Error finding container 2d37b528c25102ce628de85ce4f55e6d4fec1b9cfc69212efd0f9a539a8b856f: Status 404 returned error can't find the container with id 2d37b528c25102ce628de85ce4f55e6d4fec1b9cfc69212efd0f9a539a8b856f
Oct 30 11:48:03.839387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.839363    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"]
Oct 30 11:48:03.840345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.840306    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:03.840482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.840387    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:03.840482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:03.840429    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:04.515602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:04.515551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" event={"ID":"af2c3a63-dd7a-43cc-96a2-7186fd13763e","Type":"ContainerStarted","Data":"c0e6f38c8d0815ae0faec50a9a5883d3986f1a7b9cca45910b2e3d19eada0285"}
Oct 30 11:48:04.516140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:04.515621    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" event={"ID":"af2c3a63-dd7a-43cc-96a2-7186fd13763e","Type":"ContainerStarted","Data":"2d37b528c25102ce628de85ce4f55e6d4fec1b9cfc69212efd0f9a539a8b856f"}
Oct 30 11:48:04.575591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:04.575535    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podStartSLOduration=1.575521752 podStartE2EDuration="1.575521752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:04.574084195 +0000 UTC m=+1088.368968714" watchObservedRunningTime="2026-10-30 11:48:04.575521752 +0000 UTC m=+1088.370406306"
Oct 30 11:48:04.750299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:04.750264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:05.516437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:05.516402    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:05.518510 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:05.518475    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:05.518668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:05.518531    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:06.521895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:06.521848    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:06.522575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:06.522532    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:06.696390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:06.696351    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:07.516929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:07.516884    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:07.517242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:07.517207    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:08.523062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:08.522599    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:08.523062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:08.522674    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:08.887714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:08.887604    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:09.321790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.321729    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:09.374082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.374038    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"]
Oct 30 11:48:09.379556 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.379527    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.432185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.431925    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"]
Oct 30 11:48:09.537311 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.535504    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541610    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541653    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541726    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbdt\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541760    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541792    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541858    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541920    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.541186    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:09.554697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.542132    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:09.643523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.643424    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.644193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.644176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.644558 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.644540    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.644702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.644598    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.644842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.644825    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.645031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.645013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbdt\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.645186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.645170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.645333 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.645318    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.645523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.645508    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.647038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.647005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.647610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.647194    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.647824 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.645386    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.647928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.647236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.648018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.644067    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.654699 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.654670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.655936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.655912    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.674350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.674237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbdt\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.676146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.676121    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.827002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.826965    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:09.887486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:09.877509    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:10.214421 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:10.214378    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8c3306_98ff_4d83_a576_78aba00c24ee.slice/crio-5bd9d590acb2349f36d888329e2a06c9e89518f62031ccce3534c7c2b57bf56e WatchSource:0}: Error finding container 5bd9d590acb2349f36d888329e2a06c9e89518f62031ccce3534c7c2b57bf56e: Status 404 returned error can't find the container with id 5bd9d590acb2349f36d888329e2a06c9e89518f62031ccce3534c7c2b57bf56e
Oct 30 11:48:10.214833 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.214804    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"]
Oct 30 11:48:10.217074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.217039    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:10.217215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.217117    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:10.217215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.217163    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:10.519533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.518791    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:10.519533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.518845    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:10.699478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.697220    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" event={"ID":"3e8c3306-98ff-4d83-a576-78aba00c24ee","Type":"ContainerStarted","Data":"a63a7dc87ffe3dbaef62234702bbbf27d0501c2665e679dba0b0842c9ec0f243"}
Oct 30 11:48:10.699478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.697293    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" event={"ID":"3e8c3306-98ff-4d83-a576-78aba00c24ee","Type":"ContainerStarted","Data":"5bd9d590acb2349f36d888329e2a06c9e89518f62031ccce3534c7c2b57bf56e"}
Oct 30 11:48:10.758978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.758505    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podStartSLOduration=1.758485212 podStartE2EDuration="1.758485212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:10.750209465 +0000 UTC m=+1094.545093995" watchObservedRunningTime="2026-10-30 11:48:10.758485212 +0000 UTC m=+1094.553369707"
Oct 30 11:48:10.828065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.827983    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:10.921968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.921931    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"]
Oct 30 11:48:10.922212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.922188    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" containerID="cri-o://163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4" gracePeriod=30
Oct 30 11:48:10.936260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.927790    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"]
Oct 30 11:48:10.936260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:10.927991    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy" containerID="cri-o://fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb" gracePeriod=30
Oct 30 11:48:11.515907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:11.515861    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:48:11.516107 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:11.515929    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 11:48:11.829540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:11.829433    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 11:48:11.829540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:11.829496    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 11:48:11.867318 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:11.867287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:12.827615 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:12.827576    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 11:48:12.827781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:12.827644    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 11:48:13.561635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:13.561595    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:48:13.562128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:13.561670    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:13.796412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:13.796374    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:13.828040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:13.827888    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 11:48:13.828412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:13.828376    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 11:48:14.001942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:14.001892    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:14.004013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:14.003990    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:14.007789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:14.007767    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:48:14.827826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:14.827788    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 11:48:14.828244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:14.827856    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 11:48:15.802106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:15.801948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:15.827675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:15.827637    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 11:48:15.827878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:15.827708    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 11:48:15.891884 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:15.891846    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:48:15.892179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:15.892145    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:48:16.271325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.271294    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:48:16.320556 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320517    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320715 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320584    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320715 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320687    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320806    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320877 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320848    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320891    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.320945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320936    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.321744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.320992    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy\") pod \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\" (UID: \"a1232d7f-1026-4ff6-942f-aa8c210ccd42\") "
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.322875    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.323186    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.323398    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.323528    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.323730    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.325207    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.325367    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.327595    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:16.329143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.327795    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d" pod "a1232d7f-1026-4ff6-942f-aa8c210ccd42" (UID: "a1232d7f-1026-4ff6-942f-aa8c210ccd42"). InnerVolumeSpecName "kube-api-access-h9h7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:16.360598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.360560    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:48:16.421840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.421741    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.421840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.421827    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.421877    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.421911    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.421970    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422025    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422059    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422093    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422096    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422124    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj\") pod \"069c84b3-658d-4e00-a7f1-aebbe5173974\" (UID: \"069c84b3-658d-4e00-a7f1-aebbe5173974\") "
Oct 30 11:48:16.422503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422325    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.422503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422258    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:48:16.422616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422576    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422615    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422630    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422639    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422648    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9h7d\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-kube-api-access-h9h7d\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422658    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069c84b3-658d-4e00-a7f1-aebbe5173974-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422667    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.422678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422678    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.423019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422694    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.423019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422709    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.423019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422754    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1232d7f-1026-4ff6-942f-aa8c210ccd42-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.423019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422770    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.423019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.422778    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1232d7f-1026-4ff6-942f-aa8c210ccd42-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.423302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.423052    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.424781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.424743    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "kube-api-access-ktnzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:16.424904 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.424843    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:16.425090 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.425072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:16.425160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.425087    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo" pod "069c84b3-658d-4e00-a7f1-aebbe5173974" (UID: "069c84b3-658d-4e00-a7f1-aebbe5173974"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:48:16.523482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.523433    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.523482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.523482    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.523482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.523492    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.523798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.523500    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.523798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.523510    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnzj\" (UniqueName: \"kubernetes.io/projected/069c84b3-658d-4e00-a7f1-aebbe5173974-kube-api-access-ktnzj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.523798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.523519    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069c84b3-658d-4e00-a7f1-aebbe5173974-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:16.759196 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.759160    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerID="fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb" exitCode=0
Oct 30 11:48:16.759411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.759212    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"
Oct 30 11:48:16.759411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.759237    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" event={"ID":"a1232d7f-1026-4ff6-942f-aa8c210ccd42","Type":"ContainerDied","Data":"fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb"}
Oct 30 11:48:16.759411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.759301    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct" event={"ID":"a1232d7f-1026-4ff6-942f-aa8c210ccd42","Type":"ContainerDied","Data":"ae054a9f9973fd089985f219cbe055a1aa5fe5c628db2189199be34460adf4d2"}
Oct 30 11:48:16.759411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.759331    5060 scope.go:122] "RemoveContainer" containerID="fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb"
Oct 30 11:48:16.762668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.762324    5060 generic.go:356] "Generic (PLEG): container finished" podID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerID="163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4" exitCode=0
Oct 30 11:48:16.762668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.762510    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" event={"ID":"069c84b3-658d-4e00-a7f1-aebbe5173974","Type":"ContainerDied","Data":"163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4"}
Oct 30 11:48:16.762668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.762563    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm" event={"ID":"069c84b3-658d-4e00-a7f1-aebbe5173974","Type":"ContainerDied","Data":"f9fc6fdf4811e17d2f19d8aca5aec14cc31a09a9969b760e63df6278490517bf"}
Oct 30 11:48:16.762668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.762648    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"
Oct 30 11:48:16.776425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.776403    5060 scope.go:122] "RemoveContainer" containerID="fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb"
Oct 30 11:48:16.776892 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:48:16.776865    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb\": container with ID starting with fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb not found: ID does not exist" containerID="fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb"
Oct 30 11:48:16.777005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.776904    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb"} err="failed to get container status \"fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb\": rpc error: code = NotFound desc = could not find container \"fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb\": container with ID starting with fa822cb092037882c44351ba133ac51eb732f80c0f5dc304ff94e7683bd7f5fb not found: ID does not exist"
Oct 30 11:48:16.777005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.776930    5060 scope.go:122] "RemoveContainer" containerID="163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4"
Oct 30 11:48:16.788074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.788050    5060 scope.go:122] "RemoveContainer" containerID="c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"
Oct 30 11:48:16.800293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.800267    5060 scope.go:122] "RemoveContainer" containerID="163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4"
Oct 30 11:48:16.800588 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:48:16.800565    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4\": container with ID starting with 163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4 not found: ID does not exist" containerID="163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4"
Oct 30 11:48:16.800683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.800597    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4"} err="failed to get container status \"163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4\": rpc error: code = NotFound desc = could not find container \"163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4\": container with ID starting with 163c1ffd6dcffd3852460f2d62d34e0e78648e3aa88ad974313142ca6664dba4 not found: ID does not exist"
Oct 30 11:48:16.800683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.800623    5060 scope.go:122] "RemoveContainer" containerID="c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"
Oct 30 11:48:16.800941 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:48:16.800917    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf\": container with ID starting with c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf not found: ID does not exist" containerID="c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"
Oct 30 11:48:16.801013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.800950    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf"} err="failed to get container status \"c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf\": rpc error: code = NotFound desc = could not find container \"c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf\": container with ID starting with c22f5c8bda79465d0f3289c025891a9478dd5393e8a9e1853edf6b865e13dbdf not found: ID does not exist"
Oct 30 11:48:16.817724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.817693    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"]
Oct 30 11:48:16.828093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.828054    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 11:48:16.828592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.828124    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 11:48:16.829311 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.829291    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmtjm"]
Oct 30 11:48:16.860566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.860532    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"]
Oct 30 11:48:16.864276 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:16.864243    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnsct"]
Oct 30 11:48:17.679036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:17.679000    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:17.866526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:17.866493    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:17.867968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:17.867901    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:17.869302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:17.869279    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:48:18.335686 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:18.335653    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" path="/var/lib/kubelet/pods/069c84b3-658d-4e00-a7f1-aebbe5173974/volumes"
Oct 30 11:48:18.336336 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:18.336318    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" path="/var/lib/kubelet/pods/a1232d7f-1026-4ff6-942f-aa8c210ccd42/volumes"
Oct 30 11:48:19.651988 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:19.651944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:19.864905 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:48:19.864870    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:48:21.498700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.498660    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:21.858012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.857928    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"]
Oct 30 11:48:21.858717 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858699    5060 cpu_manager.go:414] "containerMap: removing container" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy"
Oct 30 11:48:21.858821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858718    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy"
Oct 30 11:48:21.858821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858751    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy"
Oct 30 11:48:21.858821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858757    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy"
Oct 30 11:48:21.858984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858849    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy"
Oct 30 11:48:21.858984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858858    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy"
Oct 30 11:48:21.858984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.858867    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1232d7f-1026-4ff6-942f-aa8c210ccd42" containerName="istio-proxy"
Oct 30 11:48:21.864154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.864128    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.878552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.878523    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"]
Oct 30 11:48:21.886328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.886299    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"distinct-2-w4-849669-kserve-self-signed-certs\""
Oct 30 11:48:21.890864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.890831    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.891003 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.890874    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.891049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.891020    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.891181 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.891161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htbl\" (UniqueName: \"kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.891229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.891215    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.891272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.891257    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:21.945936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.945897    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:22.000203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.999218    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.000203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.999294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.000203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.999387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.000203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.999544    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htbl\" (UniqueName: \"kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.000203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.999598    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.000203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:21.999638    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.006232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.006198    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.006538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.006312    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.006678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.006420    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.012889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.012861    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.022302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.022272    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.031808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.031776    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htbl\" (UniqueName: \"kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl\") pod \"distinct-2-w4-849669-kserve-55ff955464-nxsgh\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") " pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.181164 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.181019    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:22.503883 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:22.503855    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095a98a9_dc08_42d0_8ef6_5bdb796c2c54.slice/crio-ba4c32f5ef45014e0013435677274fd3c9d466c49a025eea042605527b5021e6 WatchSource:0}: Error finding container ba4c32f5ef45014e0013435677274fd3c9d466c49a025eea042605527b5021e6: Status 404 returned error can't find the container with id ba4c32f5ef45014e0013435677274fd3c9d466c49a025eea042605527b5021e6
Oct 30 11:48:22.504253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.504232    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"]
Oct 30 11:48:22.814630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.814520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" event={"ID":"095a98a9-dc08-42d0-8ef6-5bdb796c2c54","Type":"ContainerStarted","Data":"1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084"}
Oct 30 11:48:22.814630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.814567    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" event={"ID":"095a98a9-dc08-42d0-8ef6-5bdb796c2c54","Type":"ContainerStarted","Data":"ba4c32f5ef45014e0013435677274fd3c9d466c49a025eea042605527b5021e6"}
Oct 30 11:48:22.850948 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.850910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:22.872242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:22.872183    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podStartSLOduration=1.872167123 podStartE2EDuration="1.872167123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:22.869132578 +0000 UTC m=+1106.664017073" watchObservedRunningTime="2026-10-30 11:48:22.872167123 +0000 UTC m=+1106.667051617"
Oct 30 11:48:23.043917 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:23.043866    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:23.182087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:23.182002    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:23.654779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:23.654709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:24.682488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:24.675562    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:24.852476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:24.852259    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:26.222476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:26.174079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:26.718089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:26.718053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:27.156538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.155551    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 11:48:27.435541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.431874    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.480517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.476573    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:48:27.480517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.476669    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.531605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.531558    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.763233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.763193    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:27.795057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:27.795010    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:28.254883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:28.254663    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:28.269121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:28.269079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:28.435484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:28.435422    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 11:48:28.588245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:28.588158    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:48:28.667719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:28.667687    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:48:29.977646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:29.977598    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:31.851578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:31.851532    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:32.181572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:32.181489    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:32.193324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:32.193289    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:32.195166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:32.195143    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:32.339604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:32.339564    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:32.631442 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:32.631414    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:48:33.326037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.326001    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"]
Oct 30 11:48:33.326598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.326538    5060 cpu_manager.go:414] "containerMap: removing container" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy"
Oct 30 11:48:33.326598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.326551    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069c84b3-658d-4e00-a7f1-aebbe5173974" containerName="istio-proxy"
Oct 30 11:48:33.331311 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.331290    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.345504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.345477    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"]
Oct 30 11:48:33.350820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.350792    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.359963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.359933    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"]
Oct 30 11:48:33.379146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.379116    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"]
Oct 30 11:48:33.471523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471486    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.471523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471526    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2md\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.471823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471618    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.471823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471657    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.471823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.471823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471737    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.471823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471804    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxdz\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.472126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471881    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.472126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471917    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.472126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.472126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.471979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.472308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472151    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.472308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472195    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.472308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472283    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.472438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472337    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.472438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472359    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.472546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472494    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.472546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.472522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573398    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573479    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.573646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573503    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573525    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.573646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573550    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.573646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573609    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573698    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573879    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2md\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573897    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573939    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573965    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.573989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.573989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574042    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxdz\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574113    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574150    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574180    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574210    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.574496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.575009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.575009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.575009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.574977    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.575247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.575209    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.575247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.575232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.575417 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.575308    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.575668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.575645    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.576795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.576726    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.577396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.577361    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.577396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.577374    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.577747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.577729    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.598688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.598624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2md\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.599140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.599114    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.599245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.599210    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.599311 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.599243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxdz\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.649284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.649246    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:33.670486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.670209    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:33.708995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.708936    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:33.859369 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:33.859227    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:34.000802 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:34.000771    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b61c36_363f_4776_8822_2b6b8cbe04de.slice/crio-42014390ad77a11aadaeba75510e58eaa4e211c7f6557eb08264f8c8d3ae8d5f WatchSource:0}: Error finding container 42014390ad77a11aadaeba75510e58eaa4e211c7f6557eb08264f8c8d3ae8d5f: Status 404 returned error can't find the container with id 42014390ad77a11aadaeba75510e58eaa4e211c7f6557eb08264f8c8d3ae8d5f
Oct 30 11:48:34.001335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.001268    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"]
Oct 30 11:48:34.003212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.003179    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:34.003320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.003251    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:34.003320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.003297    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:34.020048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.020017    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"]
Oct 30 11:48:34.022149 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:34.022099    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc77535_bdd2_46bf_b536_d32ae91d3043.slice/crio-adb5fabcd9959fea550b84595c501b51f859f77acd88ac684d4e98d68d1c4d1a WatchSource:0}: Error finding container adb5fabcd9959fea550b84595c501b51f859f77acd88ac684d4e98d68d1c4d1a: Status 404 returned error can't find the container with id adb5fabcd9959fea550b84595c501b51f859f77acd88ac684d4e98d68d1c4d1a
Oct 30 11:48:34.025899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.025866    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:34.025975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.025943    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:34.026021 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.025980    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:34.042736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.042694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:34.302038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.301968    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:34.639758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.639660    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" event={"ID":"70b61c36-363f-4776-8822-2b6b8cbe04de","Type":"ContainerStarted","Data":"806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68"}
Oct 30 11:48:34.639758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.639734    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" event={"ID":"70b61c36-363f-4776-8822-2b6b8cbe04de","Type":"ContainerStarted","Data":"42014390ad77a11aadaeba75510e58eaa4e211c7f6557eb08264f8c8d3ae8d5f"}
Oct 30 11:48:34.641820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.641791    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" event={"ID":"cdc77535-bdd2-46bf-b536-d32ae91d3043","Type":"ContainerStarted","Data":"2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9"}
Oct 30 11:48:34.641951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.641845    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" event={"ID":"cdc77535-bdd2-46bf-b536-d32ae91d3043","Type":"ContainerStarted","Data":"adb5fabcd9959fea550b84595c501b51f859f77acd88ac684d4e98d68d1c4d1a"}
Oct 30 11:48:34.649848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.649818    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:34.670617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.670587    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:34.718751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.718686    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podStartSLOduration=1.718665561 podStartE2EDuration="1.718665561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:34.697996059 +0000 UTC m=+1118.492880575" watchObservedRunningTime="2026-10-30 11:48:34.718665561 +0000 UTC m=+1118.513550059"
Oct 30 11:48:34.756712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:34.756624    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podStartSLOduration=1.756599951 podStartE2EDuration="1.756599951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:34.750212459 +0000 UTC m=+1118.545096955" watchObservedRunningTime="2026-10-30 11:48:34.756599951 +0000 UTC m=+1118.551484448"
Oct 30 11:48:35.651038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:35.650789    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:48:35.651038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:35.650846    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 11:48:35.675082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:35.675041    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 11:48:35.675316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:35.675297    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 11:48:35.754396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:35.754355    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:36.096480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:36.086839    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:36.275569 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:36.267200    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:36.649653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:36.649617    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:48:36.649843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:36.649664    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 11:48:36.670756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:36.670723    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 11:48:36.671232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:36.670780    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 11:48:37.650469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:37.650419    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:48:37.650672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:37.650487    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 11:48:37.670988 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:37.670953    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 11:48:37.671496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:37.671028    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 11:48:37.845466 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:37.845402    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:38.649696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.649661    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:48:38.649873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.649716    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 11:48:38.670784 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.670683    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 11:48:38.670784 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.670742    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 11:48:38.916580 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.903272    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"]
Oct 30 11:48:38.942480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.942429    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"]
Oct 30 11:48:38.942758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.942575    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:38.957945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.957923    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:48:38.970609 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.970578    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:48:38.971687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.971666    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:48:38.971811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:38.971679    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:48:39.022515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.022442    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8wq\" (UniqueName: \"kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.022954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.022931    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.023344 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.023323    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.133277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.133232    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.133657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.133632    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8wq\" (UniqueName: \"kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.133886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.133870    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.134996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.134972    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.144409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.144091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.161184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.161158    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8wq\" (UniqueName: \"kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq\") pod \"maas-api-revoke-sub-848ac2b9-v7sbn\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.263998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.263966    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:39.650350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.650244    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:48:39.650546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.650346    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 11:48:39.660128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.660100    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"]
Oct 30 11:48:39.667142 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:39.667113    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e22cf4_237e_4b04_8818_aca99a25e554.slice/crio-4bb4d11323b5961c4e230164dbda0e9d5cf7d88297266a540bcf6d31fe78bfc4 WatchSource:0}: Error finding container 4bb4d11323b5961c4e230164dbda0e9d5cf7d88297266a540bcf6d31fe78bfc4: Status 404 returned error can't find the container with id 4bb4d11323b5961c4e230164dbda0e9d5cf7d88297266a540bcf6d31fe78bfc4
Oct 30 11:48:39.670607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.670586    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 11:48:39.670685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.670631    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 11:48:39.743520 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.743402    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn" event={"ID":"58e22cf4-237e-4b04-8818-aca99a25e554","Type":"ContainerStarted","Data":"4bb4d11323b5961c4e230164dbda0e9d5cf7d88297266a540bcf6d31fe78bfc4"}
Oct 30 11:48:39.819476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:39.818617    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:40.650636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:40.650592    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:48:40.651085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:40.650668    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 11:48:40.671483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:40.671015    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 11:48:40.671483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:40.671083    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 11:48:40.754119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:40.754069    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn" event={"ID":"58e22cf4-237e-4b04-8818-aca99a25e554","Type":"ContainerStarted","Data":"2d248ebccdc29c41c140dd63793087bc38df2b5b5294881e230da6825d95824b"}
Oct 30 11:48:40.798682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:40.798610    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn" podStartSLOduration=2.798588764 podStartE2EDuration="2.798588764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:40.794961794 +0000 UTC m=+1124.589846286" watchObservedRunningTime="2026-10-30 11:48:40.798588764 +0000 UTC m=+1124.593473261"
Oct 30 11:48:41.081290 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.081254    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"]
Oct 30 11:48:41.089817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.089148    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.161945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.161893    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"]
Oct 30 11:48:41.176052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.175937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176048    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176164    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176205    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176295 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176237    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpw4\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176324    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176362    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.176701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.176409    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277336 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277298    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277365    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277473    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277502    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpw4\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277578    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.277987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.277729    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.278154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.278125    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.278792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.278766    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.279001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.278978    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.279126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.278996    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.279513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.279480    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.282186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.282152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.282673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.282649    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.304322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.304282    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.305323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.305297    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpw4\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tcnb5\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.425441 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.425175    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:41.818319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.818258    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"]
Oct 30 11:48:41.819433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.819405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:41.835501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.835444    5060 generic.go:356] "Generic (PLEG): container finished" podID="58e22cf4-237e-4b04-8818-aca99a25e554" containerID="2d248ebccdc29c41c140dd63793087bc38df2b5b5294881e230da6825d95824b" exitCode=0
Oct 30 11:48:41.835683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.835581    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn" event={"ID":"58e22cf4-237e-4b04-8818-aca99a25e554","Type":"ContainerDied","Data":"2d248ebccdc29c41c140dd63793087bc38df2b5b5294881e230da6825d95824b"}
Oct 30 11:48:41.842073 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:41.842034    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96900fe1_8b8d_4c32_895b_3fefda261c4e.slice/crio-99ee79b8cccd1cd9f3eacbd020144379f523c2d6b5171744f3c7e31b04e57f5d WatchSource:0}: Error finding container 99ee79b8cccd1cd9f3eacbd020144379f523c2d6b5171744f3c7e31b04e57f5d: Status 404 returned error can't find the container with id 99ee79b8cccd1cd9f3eacbd020144379f523c2d6b5171744f3c7e31b04e57f5d
Oct 30 11:48:41.854492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.853102    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:41.854492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.853197    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:41.854492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:41.853242    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:42.029882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.028382    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:42.030063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.030005    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:42.031504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.031441    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:48:42.042949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.042916    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:42.043566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.043514    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:42.044574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.044544    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:48:42.845552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.845513    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" event={"ID":"96900fe1-8b8d-4c32-895b-3fefda261c4e","Type":"ContainerStarted","Data":"3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b"}
Oct 30 11:48:42.845983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.845597    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" event={"ID":"96900fe1-8b8d-4c32-895b-3fefda261c4e","Type":"ContainerStarted","Data":"99ee79b8cccd1cd9f3eacbd020144379f523c2d6b5171744f3c7e31b04e57f5d"}
Oct 30 11:48:42.917476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:42.916509    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podStartSLOduration=1.915466538 podStartE2EDuration="1.915466538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:42.913038223 +0000 UTC m=+1126.707922719" watchObservedRunningTime="2026-10-30 11:48:42.915466538 +0000 UTC m=+1126.710351216"
Oct 30 11:48:43.057619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.057591    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:43.172640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.172531    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca\") pod \"58e22cf4-237e-4b04-8818-aca99a25e554\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") "
Oct 30 11:48:43.172819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.172661    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq\" (UniqueName: \"kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq\") pod \"58e22cf4-237e-4b04-8818-aca99a25e554\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") "
Oct 30 11:48:43.172884 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.172815    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca\") pod \"58e22cf4-237e-4b04-8818-aca99a25e554\" (UID: \"58e22cf4-237e-4b04-8818-aca99a25e554\") "
Oct 30 11:48:43.172938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.172896    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca" pod "58e22cf4-237e-4b04-8818-aca99a25e554" (UID: "58e22cf4-237e-4b04-8818-aca99a25e554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:48:43.173180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.173164    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:43.175069 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.175038    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca" pod "58e22cf4-237e-4b04-8818-aca99a25e554" (UID: "58e22cf4-237e-4b04-8818-aca99a25e554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:48:43.206856 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.195726    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq" pod "58e22cf4-237e-4b04-8818-aca99a25e554" (UID: "58e22cf4-237e-4b04-8818-aca99a25e554"). InnerVolumeSpecName "kube-api-access-8g8wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:43.274260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.274225    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8wq\" (UniqueName: \"kubernetes.io/projected/58e22cf4-237e-4b04-8818-aca99a25e554-kube-api-access-8g8wq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:43.274260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.274255    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e22cf4-237e-4b04-8818-aca99a25e554-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:48:43.426143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.426054    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:43.427762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.427733    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:43.427871 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.427793    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:43.647933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.647885    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:43.809483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.807977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:43.857381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.857338    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"
Oct 30 11:48:43.857381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.857349    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn" event={"ID":"58e22cf4-237e-4b04-8818-aca99a25e554","Type":"ContainerDied","Data":"4bb4d11323b5961c4e230164dbda0e9d5cf7d88297266a540bcf6d31fe78bfc4"}
Oct 30 11:48:43.857848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:43.857420    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bb4d11323b5961c4e230164dbda0e9d5cf7d88297266a540bcf6d31fe78bfc4"
Oct 30 11:48:44.430506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:44.430470    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:44.430811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:44.430785    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:45.427082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:45.427043    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:45.427495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:45.427121    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:45.607798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:45.606983    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:46.427010 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:46.426977    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:46.427201 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:46.427049    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:47.427131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:47.427091    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:47.427288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:47.427162    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:47.537484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:47.537417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:48.427514 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:48.427473    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:48.428152 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:48.428112    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:48.518878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:48.518840    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:48.676444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:48.676408    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:49.431480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:49.431111    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:49.431480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:49.431203    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:49.628491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:49.626851    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:50.079679 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:48:50.079638    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:48:50.426633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:50.426521    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 11:48:50.426633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:50.426594    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 11:48:51.169954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:51.169917    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:51.671883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:51.671849    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:51.675914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:51.675891    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:51.676843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:51.676817    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:48:53.354102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:53.354070    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:55.058778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:55.058735    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:55.355383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:55.355277    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:55.504244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:55.504207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:56.129044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.129000    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"]
Oct 30 11:48:56.129621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.129606    5060 cpu_manager.go:414] "containerMap: removing container" podUID="58e22cf4-237e-4b04-8818-aca99a25e554" containerName="revoke-keys"
Oct 30 11:48:56.129676 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.129623    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e22cf4-237e-4b04-8818-aca99a25e554" containerName="revoke-keys"
Oct 30 11:48:56.129715 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.129701    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e22cf4-237e-4b04-8818-aca99a25e554" containerName="revoke-keys"
Oct 30 11:48:56.134590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.134563    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.168037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.168003    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"]
Oct 30 11:48:56.275034 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.274991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275053    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8sc9\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275144    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275191    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275255    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275320    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275442 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275375    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275442 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275401    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.275442 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.275427    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376256    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8sc9\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376313    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376377    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376436    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376517    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.376996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376963    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.377285 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.376966    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.377285 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.377254    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.377515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.377488    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.377779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.377751    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.379336 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.379275    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.380086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.380057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.401059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.401026    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8sc9\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.401059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.401038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tzctd\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.450571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.450540    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:56.610553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.610521    5060 scope.go:122] "RemoveContainer" containerID="e2c6d5256cd4efaae6b7545e0bbaa82fa01781fe9d6aa1c00013aaea64df2aa2"
Oct 30 11:48:56.620675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.620650    5060 scope.go:122] "RemoveContainer" containerID="50d7e0b3730de05d32b10b66fd07a08bd4f2531924ee2832eafa8fa77eebce72"
Oct 30 11:48:56.777988 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:48:56.777961    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69ee38c_c8c8_4c63_8ce1_5d03b443c54f.slice/crio-6c7446d575d3cce449eed9ce49a74feea302d455ed45b7d23acd87eacfc14066 WatchSource:0}: Error finding container 6c7446d575d3cce449eed9ce49a74feea302d455ed45b7d23acd87eacfc14066: Status 404 returned error can't find the container with id 6c7446d575d3cce449eed9ce49a74feea302d455ed45b7d23acd87eacfc14066
Oct 30 11:48:56.779304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.779281    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"]
Oct 30 11:48:56.781117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.781086    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:56.781218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.781158    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:56.781218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:56.781203    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:48:57.078060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:57.078017    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" event={"ID":"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f","Type":"ContainerStarted","Data":"c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5"}
Oct 30 11:48:57.078272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:57.078084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" event={"ID":"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f","Type":"ContainerStarted","Data":"6c7446d575d3cce449eed9ce49a74feea302d455ed45b7d23acd87eacfc14066"}
Oct 30 11:48:57.260738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:57.260680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:57.451084 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:57.451056    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:48:57.452887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:57.452862    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:48:57.453029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:57.452919    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:48:58.451020 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:58.450984    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:48:58.451497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:58.451035    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:48:59.211720 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:59.211681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:48:59.451048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:59.451010    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:48:59.451597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:48:59.451064    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:49:00.451654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:00.451618    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:49:00.452074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:00.451673    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:49:01.405240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:01.405192    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:01.450984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:01.450950    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:49:01.451166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:01.451009    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:49:01.560545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:01.560496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:02.077408 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:02.077371    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:02.331953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:02.331805    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:02.451596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:02.451553    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:49:02.451786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:02.451614    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:49:02.495331 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:02.495296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:02.739737 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:02.739695    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:03.451241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:03.451201    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:49:03.451433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:03.451285    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:49:03.639155 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:03.639120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:04.451734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:04.451701    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tzctd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 11:49:04.452316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:04.452285    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 11:49:04.508546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:04.508504    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:05.524033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:05.523985    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:49:05.524930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:05.524904    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:49:05.525682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:05.525662    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:49:05.561024 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:05.560965    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:05.596400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:05.596341    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podStartSLOduration=9.596324975 podStartE2EDuration="9.596324975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:57.136731638 +0000 UTC m=+1140.931616160" watchObservedRunningTime="2026-10-30 11:49:05.596324975 +0000 UTC m=+1149.391209470"
Oct 30 11:49:07.464375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:07.464335    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:09.153848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:09.153809    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:10.936468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:10.936385    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:11.164807 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.164772    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"]
Oct 30 11:49:11.170171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.170148    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.222193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.222156    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"]
Oct 30 11:49:11.248186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248146    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248217    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248395    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248417    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdnn\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248447    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248651 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248510    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248720 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248695    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.248785 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.248767    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.352329 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.352284    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.352646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.352619    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.352731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.352665    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.353617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.353586    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.353801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.353779    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdnn\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.353971 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.353953    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.354143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.354124    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.354330 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.354313    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.354497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.354482    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.354772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.354745    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.354863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.354027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.355264 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.355228    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.355701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.355679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.356076 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.356050    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.361196 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.361160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.363502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.363287    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.413420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.413385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdnn\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.419440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.419405    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z262d\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.485111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.485020    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:11.820482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.820441    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"]
Oct 30 11:49:11.821025 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:49:11.821001    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c1fa20_c3f7_4606_9dae_8b81572ef711.slice/crio-b6011da0b4b6a8860f2da5f8f66959a28142eb5bb9d003bf37fb20a919885da6 WatchSource:0}: Error finding container b6011da0b4b6a8860f2da5f8f66959a28142eb5bb9d003bf37fb20a919885da6: Status 404 returned error can't find the container with id b6011da0b4b6a8860f2da5f8f66959a28142eb5bb9d003bf37fb20a919885da6
Oct 30 11:49:11.823815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.823779    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:11.823918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.823852    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:11.823918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:11.823885    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:12.285845 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:12.285803    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" event={"ID":"97c1fa20-c3f7-4606-9dae-8b81572ef711","Type":"ContainerStarted","Data":"89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f"}
Oct 30 11:49:12.286511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:12.286484    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" event={"ID":"97c1fa20-c3f7-4606-9dae-8b81572ef711","Type":"ContainerStarted","Data":"b6011da0b4b6a8860f2da5f8f66959a28142eb5bb9d003bf37fb20a919885da6"}
Oct 30 11:49:12.485875 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:12.485836    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:12.487990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:12.487946    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:12.488715 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:12.488276    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:13.490943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:13.490901    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:13.491562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:13.491526    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:13.794657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:13.794151    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:13.859498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:13.858494    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:49:14.084536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.083827    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.142484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.141611    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.162483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.154989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:14.195653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.194130    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.216165 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.216133    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:49:14.216497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.216441    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.247464 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.247397    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:49:14.492480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.492329    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:14.492480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.492388    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:14.632479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.631806    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.892536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.892406    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.918484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.892816    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 11:49:14.918484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:14.893534    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:49:15.054481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.029787    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podStartSLOduration=4.029764971 podStartE2EDuration="4.029764971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:12.340540303 +0000 UTC m=+1156.135424797" watchObservedRunningTime="2026-10-30 11:49:15.029764971 +0000 UTC m=+1158.824649466"
Oct 30 11:49:15.156546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.090577    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:15.156546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.091129    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:49:15.180480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.166624    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:15.180480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.166833    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:49:15.565487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.557558    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:15.573665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.573609    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:15.592666 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.580644    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:15.850535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.845825    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:15.871491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.851285    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:49:15.872855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.872822    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:49:16.032479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.906527    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 11:49:16.055527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.906570    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 11:49:16.065481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.996713    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 11:49:16.065481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:15.996779    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:49:16.065481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.062066    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:16.109547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.095083    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:16.141047 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.095561    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:49:16.141047 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.126211    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"]
Oct 30 11:49:16.185518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.185479    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 11:49:16.229919 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.229888    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.342593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.342543    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:16.378087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.377991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.400795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.390801    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"]
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.414811    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.414892    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.414993    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.415032    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6c2p\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.415111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.415166    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.415304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.419495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.415330    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.540653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.508148    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:16.540653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.508205    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:16.543551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.543516    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.543813 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.543787    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6c2p\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.543969 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.543955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.544079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.544065    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.544254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.544238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.544363 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.544349    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.544620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.544575    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.544812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.544795    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.544946 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.544933    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.562271 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.545729    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.691925 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.549786    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.816200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.549999    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.816427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.550738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.816640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.573869    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.816781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.635618    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:16.819530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.675434    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:16.819530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.815628    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.819530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.818806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6c2p\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.850940 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.850904    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.879507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.866594    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:16.881878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.881852    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:49:16.882179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.882162    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:49:16.882312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.882298    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"]
Oct 30 11:49:16.951719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.950060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:16.955625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.953345    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"]
Oct 30 11:49:16.955625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:16.953512    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.018312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.015823    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:17.036731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.036697    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:17.045487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.041838    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:17.061482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.051410    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:49:17.151752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.062876    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 11:49:17.163479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.100570    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.186478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.163824    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.186883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.186848    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.187027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.187011    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5k8m\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.187127 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.187114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.187283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.187265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.195522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.139037    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:17.195522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.187522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.195522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.187554    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.195522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.187581    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.195522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.139081    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:17.195522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.188284    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:17.206561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.199550    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:17.206561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.199982    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:49:17.225947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.220948    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:49:17.341194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.341146    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.341515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.341492    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5k8m\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.341750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.341730    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.389334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.389520    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.389556    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.389587    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.389766    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.389849    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.395679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.395946    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.396200    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.421883    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.423864    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:49:17.448747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.424611    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:49:17.474959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.474820    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.566647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.566599    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:17.566858 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.566669    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:17.567227 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.567202    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.575597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.575569    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:49:17.576370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.576345    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.702766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.658171    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:49:17.702766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.667164    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:17.768070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:17.767971    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:18.031528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.029668    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5k8m\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:18.052479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.033298    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:18.052858 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.052815    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:49:18.277594 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.277543    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:18.382300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.382210    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:49:18.430214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.430182    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:49:18.522695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.522641    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:18.537342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.537246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:18.537629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.537557    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:18.565744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.565709    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:18.566015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.565983    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:18.667522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.667423    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:49:18.667679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.667552    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:49:18.667679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.667585    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"]
Oct 30 11:49:18.740276 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.739506    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:49:18.740276 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.739548    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"]
Oct 30 11:49:18.740871 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.740390    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:18.767296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.767124    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:49:18.767296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.767251    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:49:18.820599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.819713    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:18.843492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.839669    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:49:18.880662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.840723    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:49:18.880882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.842839    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:18.881249 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.881224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpjk\" (UniqueName: \"kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:18.881543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.881525    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.011481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:18.988420    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.011481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.010040    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpjk\" (UniqueName: \"kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.011481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.010174    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.054787 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.012069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.079861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.058263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.079861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.074332    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpjk\" (UniqueName: \"kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk\") pod \"maas-api-revoke-sub-2dc16095-cpztl\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.157486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.155789    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:19.223305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.222206    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"]
Oct 30 11:49:19.226108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.226026    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:19.226376 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.226151    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:19.226376 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.226195    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:19.237994 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.237960    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:19.319373 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:49:19.319330    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b748992_0b9d_41ce_9986_83f3ed5296f7.slice/crio-dcd460c6e64ca20b1e91d716b2d5e1b40ff8ecd1a80d393e92ead0e486dc0fb2 WatchSource:0}: Error finding container dcd460c6e64ca20b1e91d716b2d5e1b40ff8ecd1a80d393e92ead0e486dc0fb2: Status 404 returned error can't find the container with id dcd460c6e64ca20b1e91d716b2d5e1b40ff8ecd1a80d393e92ead0e486dc0fb2
Oct 30 11:49:19.320567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.320432    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"]
Oct 30 11:49:19.324338 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.324291    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:19.324480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.324371    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:19.324480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.324400    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:49:19.487817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.487693    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:19.487817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.487755    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:19.565884 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.565839    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" event={"ID":"4b748992-0b9d-41ce-9986-83f3ed5296f7","Type":"ContainerStarted","Data":"224f9160ba3c067a0b2ffd5a69f1b74c8d651039900eb77b92b7ac755614b56a"}
Oct 30 11:49:19.566093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.565916    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" event={"ID":"4b748992-0b9d-41ce-9986-83f3ed5296f7","Type":"ContainerStarted","Data":"dcd460c6e64ca20b1e91d716b2d5e1b40ff8ecd1a80d393e92ead0e486dc0fb2"}
Oct 30 11:49:19.571973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.571809    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" event={"ID":"ab5ceff7-b397-46cd-a04f-4aeb01146dce","Type":"ContainerStarted","Data":"a40d799a8bce1b156551a1d9f4252fc3a0470abb3e4aacac01983ff2fd495da7"}
Oct 30 11:49:19.571973 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.571879    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" event={"ID":"ab5ceff7-b397-46cd-a04f-4aeb01146dce","Type":"ContainerStarted","Data":"56abb5a8cdf58b8abcd8961daeaac196cbfffdd847051127ccbe43ae781b9110"}
Oct 30 11:49:19.636401 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.636359    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podStartSLOduration=4.636343335 podStartE2EDuration="4.636343335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:19.618101414 +0000 UTC m=+1163.412985922" watchObservedRunningTime="2026-10-30 11:49:19.636343335 +0000 UTC m=+1163.431227828"
Oct 30 11:49:19.636874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.636515    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"]
Oct 30 11:49:19.671524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:19.669409    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podStartSLOduration=4.669390185 podStartE2EDuration="4.669390185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:19.668002612 +0000 UTC m=+1163.462887108" watchObservedRunningTime="2026-10-30 11:49:19.669390185 +0000 UTC m=+1163.464274678"
Oct 30 11:49:19.866504 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:49:19.866404    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:49:20.046357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.045189    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:20.046587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.046396    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:20.046587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.046484    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:20.053694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.053662    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:20.055121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.055088    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:20.055257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.055138    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:20.491538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.490201    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:20.491538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.490280    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:20.572825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.572796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:20.654475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.651065    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl" event={"ID":"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716","Type":"ContainerStarted","Data":"833c12e20926fbd799e9eee4a77751d212f4cfb4191e55631ff6d09b125738ce"}
Oct 30 11:49:20.655652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.654725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl" event={"ID":"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716","Type":"ContainerStarted","Data":"ee8d2fd8cfe8f4c14e1b7edddd719211f9dab16402ce86e9f5d66670f570e544"}
Oct 30 11:49:20.731473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:20.731015    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl" podStartSLOduration=3.730995549 podStartE2EDuration="3.730995549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:20.695526955 +0000 UTC m=+1164.490411450" watchObservedRunningTime="2026-10-30 11:49:20.730995549 +0000 UTC m=+1164.525880045"
Oct 30 11:49:21.047093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.047055    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:21.047724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.047689    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:21.054043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.053913    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:21.054043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.053984    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:21.488480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.488311    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:21.488480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.488372    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:21.659476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.658651    5060 generic.go:356] "Generic (PLEG): container finished" podID="bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" containerID="833c12e20926fbd799e9eee4a77751d212f4cfb4191e55631ff6d09b125738ce" exitCode=0
Oct 30 11:49:21.659476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:21.658832    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl" event={"ID":"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716","Type":"ContainerDied","Data":"833c12e20926fbd799e9eee4a77751d212f4cfb4191e55631ff6d09b125738ce"}
Oct 30 11:49:22.045089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.045050    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:22.045289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.045109    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:22.053815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.053768    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:22.054325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.053829    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:22.360587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.360484    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:22.523480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.504900    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 11:49:22.523480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:22.504964    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 11:49:23.088482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.056552    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:23.088482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.056621    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:23.171334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.170592    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:23.171334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.170651    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:23.339531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.339221    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:23.531771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.531717    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca\") pod \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") "
Oct 30 11:49:23.532212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.532191    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca\") pod \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") "
Oct 30 11:49:23.532523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.532496    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk\" (UniqueName: \"kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk\") pod \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\" (UID: \"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716\") "
Oct 30 11:49:23.546253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.546215    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca" pod "bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" (UID: "bb07c9ba-8eae-4fb7-ac57-e565ebcc5716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:49:23.550670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.550610    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca" pod "bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" (UID: "bb07c9ba-8eae-4fb7-ac57-e565ebcc5716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:49:23.559360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.559289    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk" pod "bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" (UID: "bb07c9ba-8eae-4fb7-ac57-e565ebcc5716"). InnerVolumeSpecName "kube-api-access-xcpjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:49:23.645058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.644904    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:49:23.645058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.644946    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcpjk\" (UniqueName: \"kubernetes.io/projected/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-kube-api-access-xcpjk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:49:23.645058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.644965    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:49:23.710511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.705768    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl" event={"ID":"bb07c9ba-8eae-4fb7-ac57-e565ebcc5716","Type":"ContainerDied","Data":"ee8d2fd8cfe8f4c14e1b7edddd719211f9dab16402ce86e9f5d66670f570e544"}
Oct 30 11:49:23.710511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.705830    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee8d2fd8cfe8f4c14e1b7edddd719211f9dab16402ce86e9f5d66670f570e544"
Oct 30 11:49:23.710511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:23.705899    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"
Oct 30 11:49:24.044814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.044781    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:24.045060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.045037    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:24.053429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.053399    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:24.054110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.054086    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:24.270627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.270587    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:24.534719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.534651    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:49:24.535086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:24.535045    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:25.045945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:25.045902    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:25.046261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:25.046229    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:25.053216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:25.053135    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:25.053216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:25.053193    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:25.585479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:25.584663    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:49:25.585479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:25.584748    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:26.045545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.045491    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:26.045766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.045568    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:26.053983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.053946    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:26.054206 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.054009    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:26.073648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.073610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:26.485586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.485507    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:26.627654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.627596    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z262d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:49:26.628385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:26.628343    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:27.044735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:27.044649    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 11:49:27.044735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:27.044708    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 11:49:27.054253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:27.054133    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 11:49:27.054253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:27.054191    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 11:49:27.165483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:27.161361    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:27.165483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:27.165161    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:49:28.022806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.022773    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:28.049319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.049288    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:28.049743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.049722    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:28.050389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.050373    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:49:28.058031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.058010    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:28.058308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.058289    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:28.059087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:28.059070    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:49:30.000083 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:29.999334    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:31.847609 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:31.847567    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:31.997177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:31.997145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:33.934084 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:33.934046    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:34.889652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:34.889608    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:35.761105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:35.761060    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:35.914369 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:35.914331    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:37.742365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:37.742329    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:39.575947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:39.575910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:41.460801 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:41.460762    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:42.488921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:42.488880    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:42.636515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:42.636481    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:43.289182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:43.289145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:45.477528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:45.477491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:47.375490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:47.375409    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:48.641765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:48.641727    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:49.261002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:49.260963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:49.859026 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:49:49.858989    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:49:51.055577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:51.055544    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:51.903383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:51.903282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:52.853159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:52.853120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:54.648815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:54.648773    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:56.508629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:56.508598    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:49:56.511212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:56.511186    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:49:56.665210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:56.665166    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:49:58.521165 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:49:58.521126    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:00.272537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:00.272412    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:02.082688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:02.082642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:02.608903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:02.608862    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:03.212746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:03.212680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:03.391489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:03.388314    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:04.073706 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:04.073672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:05.950041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:05.949989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:07.928333 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:07.928296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:09.762377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:09.762339    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:10.032606 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:10.032517    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"]
Oct 30 11:50:10.043827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:10.043796    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bae4b2f-l5wd9"]
Oct 30 11:50:10.343719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:10.343602    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6273511-94f4-405f-a89d-e76ddcbfd903" path="/var/lib/kubelet/pods/e6273511-94f4-405f-a89d-e76ddcbfd903/volumes"
Oct 30 11:50:11.279745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.279704    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"]
Oct 30 11:50:11.280297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.280283    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" containerName="revoke-keys"
Oct 30 11:50:11.280363 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.280298    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" containerName="revoke-keys"
Oct 30 11:50:11.280415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.280396    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" containerName="revoke-keys"
Oct 30 11:50:11.285544 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.285520    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.301822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.301794    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:50:11.303652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.303624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"]
Oct 30 11:50:11.314123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.313716    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:50:11.314123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.313803    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:50:11.314123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.313806    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:50:11.468783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.468736    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.469002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.468975    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq58x\" (UniqueName: \"kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.469230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.469202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.486324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.486282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:11.570718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.570625    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.570915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.570755    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq58x\" (UniqueName: \"kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.570985 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.570942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.571718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.571693    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.573935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.573913    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.594728 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.594691    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq58x\" (UniqueName: \"kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x\") pod \"maas-api-revoke-sub-4f5d2cc6-7jdwg\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.606600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.606571    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:11.915331 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:11.915284    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cc7563_e46e_4ca4_81f5_276b739b3b5d.slice/crio-e0f1e1de81dc45d37d93b9b656d51d672370aa0b10b2cda0e8786e6163254ec0 WatchSource:0}: Error finding container e0f1e1de81dc45d37d93b9b656d51d672370aa0b10b2cda0e8786e6163254ec0: Status 404 returned error can't find the container with id e0f1e1de81dc45d37d93b9b656d51d672370aa0b10b2cda0e8786e6163254ec0
Oct 30 11:50:11.916484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:11.916430    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"]
Oct 30 11:50:12.343251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:12.343208    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg" event={"ID":"e0cc7563-e46e-4ca4-81f5-276b739b3b5d","Type":"ContainerStarted","Data":"e77e7337d8fe65b640f0a8ffaa3360442b0d23b2339f232dea7854b4bfdebd21"}
Oct 30 11:50:12.343766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:12.343279    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg" event={"ID":"e0cc7563-e46e-4ca4-81f5-276b739b3b5d","Type":"ContainerStarted","Data":"e0f1e1de81dc45d37d93b9b656d51d672370aa0b10b2cda0e8786e6163254ec0"}
Oct 30 11:50:12.388778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:12.388706    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg" podStartSLOduration=1.3886895049999999 podStartE2EDuration="1.388689505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:12.383975667 +0000 UTC m=+1216.178860183" watchObservedRunningTime="2026-10-30 11:50:12.388689505 +0000 UTC m=+1216.183573999"
Oct 30 11:50:13.351860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:13.351829    5060 generic.go:356] "Generic (PLEG): container finished" podID="e0cc7563-e46e-4ca4-81f5-276b739b3b5d" containerID="e77e7337d8fe65b640f0a8ffaa3360442b0d23b2339f232dea7854b4bfdebd21" exitCode=0
Oct 30 11:50:13.352299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:13.351916    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg" event={"ID":"e0cc7563-e46e-4ca4-81f5-276b739b3b5d","Type":"ContainerDied","Data":"e77e7337d8fe65b640f0a8ffaa3360442b0d23b2339f232dea7854b4bfdebd21"}
Oct 30 11:50:13.408541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:13.408503    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:14.538202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.538173    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:14.612121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.612012    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca\") pod \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") "
Oct 30 11:50:14.612328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.612291    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x\" (UniqueName: \"kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x\") pod \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") "
Oct 30 11:50:14.612390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.612352    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca\") pod \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\" (UID: \"e0cc7563-e46e-4ca4-81f5-276b739b3b5d\") "
Oct 30 11:50:14.612897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.612869    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca" pod "e0cc7563-e46e-4ca4-81f5-276b739b3b5d" (UID: "e0cc7563-e46e-4ca4-81f5-276b739b3b5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:50:14.614670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.614645    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca" pod "e0cc7563-e46e-4ca4-81f5-276b739b3b5d" (UID: "e0cc7563-e46e-4ca4-81f5-276b739b3b5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:50:14.614771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.614747    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x" pod "e0cc7563-e46e-4ca4-81f5-276b739b3b5d" (UID: "e0cc7563-e46e-4ca4-81f5-276b739b3b5d"). InnerVolumeSpecName "kube-api-access-kq58x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:50:14.714821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.714785    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:14.714821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.714819    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq58x\" (UniqueName: \"kubernetes.io/projected/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-kube-api-access-kq58x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:14.714821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:14.714828    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc7563-e46e-4ca4-81f5-276b739b3b5d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:15.369222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.369184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:15.375204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.375164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg" event={"ID":"e0cc7563-e46e-4ca4-81f5-276b739b3b5d","Type":"ContainerDied","Data":"e0f1e1de81dc45d37d93b9b656d51d672370aa0b10b2cda0e8786e6163254ec0"}
Oct 30 11:50:15.375422 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.375235    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f1e1de81dc45d37d93b9b656d51d672370aa0b10b2cda0e8786e6163254ec0"
Oct 30 11:50:15.375422 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.375186    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"
Oct 30 11:50:15.957833 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.957795    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"]
Oct 30 11:50:15.958428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.958413    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e0cc7563-e46e-4ca4-81f5-276b739b3b5d" containerName="revoke-keys"
Oct 30 11:50:15.958522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.958429    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cc7563-e46e-4ca4-81f5-276b739b3b5d" containerName="revoke-keys"
Oct 30 11:50:15.958566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.958542    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cc7563-e46e-4ca4-81f5-276b739b3b5d" containerName="revoke-keys"
Oct 30 11:50:15.962251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.962231    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:15.994426 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:15.994386    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"]
Oct 30 11:50:16.031598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.031563    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.031819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.031640    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.031958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.031927    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.032033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.031974    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.032033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.032004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.032260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.032236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.032359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.032280    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndlm\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.032359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.032312    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.032501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.032476    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134304    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134415    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134465    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134568    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134711    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndlm\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134771    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.134819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134808    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.135051 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.134882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.135309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.135238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.135562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.135487    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.137102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.137077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.137325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.137302    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.159004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.158974    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.159154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.159047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndlm\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.276956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.276850    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:16.609717 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:16.609672    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fad95e2_b9b8_40ba_84ac_cff5670a363c.slice/crio-9a253ef13d20aaa0628e1406d63e2fb8e189a3f18d7690ad3ab5fdcd1188d957 WatchSource:0}: Error finding container 9a253ef13d20aaa0628e1406d63e2fb8e189a3f18d7690ad3ab5fdcd1188d957: Status 404 returned error can't find the container with id 9a253ef13d20aaa0628e1406d63e2fb8e189a3f18d7690ad3ab5fdcd1188d957
Oct 30 11:50:16.612486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.612440    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"]
Oct 30 11:50:16.612568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.612436    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:50:16.612604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.612580    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:50:16.612673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:16.612656    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:50:17.215231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:17.215194    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:17.398477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:17.398221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:17.491955 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:17.491834    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" event={"ID":"7fad95e2-b9b8-40ba-84ac-cff5670a363c","Type":"ContainerStarted","Data":"89e75f6fe163cb3505290c0a727e301bffd356db63ebfe5e72820486bbf28665"}
Oct 30 11:50:17.492298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:17.492271    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" event={"ID":"7fad95e2-b9b8-40ba-84ac-cff5670a363c","Type":"ContainerStarted","Data":"9a253ef13d20aaa0628e1406d63e2fb8e189a3f18d7690ad3ab5fdcd1188d957"}
Oct 30 11:50:17.554081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:17.554021    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podStartSLOduration=2.554001381 podStartE2EDuration="2.554001381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:17.552909157 +0000 UTC m=+1221.347793652" watchObservedRunningTime="2026-10-30 11:50:17.554001381 +0000 UTC m=+1221.348885877"
Oct 30 11:50:18.287498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:18.285527    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:18.287498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:18.285597    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 11:50:18.287498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:18.285653    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 11:50:18.999486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:18.999257    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:19.158601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.158541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:19.317539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.302196    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 11:50:19.318158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.318121    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 11:50:19.427811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.427768    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"]
Oct 30 11:50:19.433610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.433574    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.434567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.434539    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"]
Oct 30 11:50:19.460965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.460936    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-58826b\""
Oct 30 11:50:19.464997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.464969    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-58826b-dockercfg-rqm6b\""
Oct 30 11:50:19.516133 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.516034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume\") pod \"payload-processing-e2e-api-b-58826b-656cf46db6-5cln6\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") " pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.516648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.516620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275kf\" (UniqueName: \"kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf\") pod \"payload-processing-e2e-api-b-58826b-656cf46db6-5cln6\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") " pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.547286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.547254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:19.626267 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.626178    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume\") pod \"payload-processing-e2e-api-b-58826b-656cf46db6-5cln6\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") " pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.626267 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.626254    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275kf\" (UniqueName: \"kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf\") pod \"payload-processing-e2e-api-b-58826b-656cf46db6-5cln6\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") " pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.626977 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.626955    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume\") pod \"payload-processing-e2e-api-b-58826b-656cf46db6-5cln6\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") " pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.654776 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.654748    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275kf\" (UniqueName: \"kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf\") pod \"payload-processing-e2e-api-b-58826b-656cf46db6-5cln6\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") " pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.697515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.678536    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"]
Oct 30 11:50:19.708259 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.708224    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"]
Oct 30 11:50:19.708502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.708355    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.724658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.724612    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:50:19.724658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.724632    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:50:19.724920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.724760    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:50:19.724920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.724877    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:50:19.727298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.727272    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.727429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.727415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.727549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.727530    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hxn\" (UniqueName: \"kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.766341 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.766304    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:19.828494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.828432    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.828727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.828609    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.828727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.828684    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hxn\" (UniqueName: \"kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.829514 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.829490    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.831368 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.831344    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.853663 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:19.853639    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hxn\" (UniqueName: \"kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn\") pod \"maas-api-revoke-sub-d9cffafa-w58dr\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:19.927489 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:50:19.924945    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:50:20.029410 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.029376    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:20.166031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.163671    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"]
Oct 30 11:50:20.302481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.302322    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 11:50:20.302481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.302394    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 11:50:20.361443 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:20.361401    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a5fe67_da11_4387_a313_e9f0391328f4.slice/crio-3ce948f4c6e512b5acb0900f2a946ae502efa07bf3273154b3b83057686b27c7 WatchSource:0}: Error finding container 3ce948f4c6e512b5acb0900f2a946ae502efa07bf3273154b3b83057686b27c7: Status 404 returned error can't find the container with id 3ce948f4c6e512b5acb0900f2a946ae502efa07bf3273154b3b83057686b27c7
Oct 30 11:50:20.362697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.362675    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"]
Oct 30 11:50:20.555772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.555721    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6" event={"ID":"d19047b8-ff68-4543-8a2a-d3abf4836a97","Type":"ContainerStarted","Data":"172427c6cabcc9157665ad7a8becb5276ca50649c488ab7d76bd8ccbe01c2f70"}
Oct 30 11:50:20.555988 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.555787    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6" event={"ID":"d19047b8-ff68-4543-8a2a-d3abf4836a97","Type":"ContainerStarted","Data":"f1567a7bf06d82363bf02a2ae3bdde037d5176d18a581a9660f5b2b72d7324ca"}
Oct 30 11:50:20.555988 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.555818    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:20.557361 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.557327    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr" event={"ID":"19a5fe67-da11-4387-a313-e9f0391328f4","Type":"ContainerStarted","Data":"3ce948f4c6e512b5acb0900f2a946ae502efa07bf3273154b3b83057686b27c7"}
Oct 30 11:50:20.614846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:20.614749    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6" podStartSLOduration=1.614728619 podStartE2EDuration="1.614728619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:20.609810736 +0000 UTC m=+1224.404695231" watchObservedRunningTime="2026-10-30 11:50:20.614728619 +0000 UTC m=+1224.409613114"
Oct 30 11:50:21.301479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.301202    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 11:50:21.301479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.301269    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 11:50:21.502870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.502770    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:21.568160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.568074    5060 generic.go:356] "Generic (PLEG): container finished" podID="19a5fe67-da11-4387-a313-e9f0391328f4" containerID="c1a69edcdc61bdb23dd739109218839d5827565708a83ae88d096caa1a7a801a" exitCode=0
Oct 30 11:50:21.568316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.568161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr" event={"ID":"19a5fe67-da11-4387-a313-e9f0391328f4","Type":"ContainerDied","Data":"c1a69edcdc61bdb23dd739109218839d5827565708a83ae88d096caa1a7a801a"}
Oct 30 11:50:21.812477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.809708    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"]
Oct 30 11:50:21.813730 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.813706    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.831653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.831620    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"]
Oct 30 11:50:21.858610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.858565    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnclr\" (UniqueName: \"kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.858822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.858656    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.858822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.858727    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.960437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.960398    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnclr\" (UniqueName: \"kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.960654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.960516    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.960654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.960592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.961142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.961111    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:21.963218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:21.963196    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:22.003490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.002258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnclr\" (UniqueName: \"kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr\") pod \"maas-api-revoke-sub-25b3ec50-rk6ss\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:22.133819 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.133726    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:22.302371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.302059    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 11:50:22.302371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.302135    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 11:50:22.471755 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:22.471710    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8183a93a_076c_4dc1_9daf_30ee1c49733b.slice/crio-888403246df7aaa8c224156024e042462315aa73be27bc82c34d3e21158dde3a WatchSource:0}: Error finding container 888403246df7aaa8c224156024e042462315aa73be27bc82c34d3e21158dde3a: Status 404 returned error can't find the container with id 888403246df7aaa8c224156024e042462315aa73be27bc82c34d3e21158dde3a
Oct 30 11:50:22.473317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.473293    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"]
Oct 30 11:50:22.606690 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.606649    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss" event={"ID":"8183a93a-076c-4dc1-9daf-30ee1c49733b","Type":"ContainerStarted","Data":"888403246df7aaa8c224156024e042462315aa73be27bc82c34d3e21158dde3a"}
Oct 30 11:50:22.769956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.769929    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:22.874679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.874636    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca\") pod \"19a5fe67-da11-4387-a313-e9f0391328f4\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") "
Oct 30 11:50:22.874878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.874784    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca\") pod \"19a5fe67-da11-4387-a313-e9f0391328f4\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") "
Oct 30 11:50:22.874878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.874846    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn\" (UniqueName: \"kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn\") pod \"19a5fe67-da11-4387-a313-e9f0391328f4\" (UID: \"19a5fe67-da11-4387-a313-e9f0391328f4\") "
Oct 30 11:50:22.875049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.875024    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca" pod "19a5fe67-da11-4387-a313-e9f0391328f4" (UID: "19a5fe67-da11-4387-a313-e9f0391328f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:50:22.875427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.875403    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:22.877160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.877131    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn" pod "19a5fe67-da11-4387-a313-e9f0391328f4" (UID: "19a5fe67-da11-4387-a313-e9f0391328f4"). InnerVolumeSpecName "kube-api-access-l9hxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:50:22.877281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.877264    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca" pod "19a5fe67-da11-4387-a313-e9f0391328f4" (UID: "19a5fe67-da11-4387-a313-e9f0391328f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:50:22.976533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.976495    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5fe67-da11-4387-a313-e9f0391328f4-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:22.976533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:22.976531    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hxn\" (UniqueName: \"kubernetes.io/projected/19a5fe67-da11-4387-a313-e9f0391328f4-kube-api-access-l9hxn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:23.277702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.277665    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 11:50:23.277892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.277721    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 11:50:23.364050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.364007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:23.617221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.617103    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss" event={"ID":"8183a93a-076c-4dc1-9daf-30ee1c49733b","Type":"ContainerStarted","Data":"550114098a6efb598c9c300d6b9155b180ab2f624e7ac82b26f9947f0d1c2322"}
Oct 30 11:50:23.618797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.618775    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"
Oct 30 11:50:23.618950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.618813    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr" event={"ID":"19a5fe67-da11-4387-a313-e9f0391328f4","Type":"ContainerDied","Data":"3ce948f4c6e512b5acb0900f2a946ae502efa07bf3273154b3b83057686b27c7"}
Oct 30 11:50:23.619226 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.618870    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce948f4c6e512b5acb0900f2a946ae502efa07bf3273154b3b83057686b27c7"
Oct 30 11:50:23.660665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:23.660597    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss" podStartSLOduration=2.660575626 podStartE2EDuration="2.660575626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:23.658895342 +0000 UTC m=+1227.453779858" watchObservedRunningTime="2026-10-30 11:50:23.660575626 +0000 UTC m=+1227.455460118"
Oct 30 11:50:24.282989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:24.282961    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:24.283491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:24.283474    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:24.284363 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:24.284339    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:50:24.629632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:24.629538    5060 generic.go:356] "Generic (PLEG): container finished" podID="8183a93a-076c-4dc1-9daf-30ee1c49733b" containerID="550114098a6efb598c9c300d6b9155b180ab2f624e7ac82b26f9947f0d1c2322" exitCode=0
Oct 30 11:50:24.630111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:24.629620    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss" event={"ID":"8183a93a-076c-4dc1-9daf-30ee1c49733b","Type":"ContainerDied","Data":"550114098a6efb598c9c300d6b9155b180ab2f624e7ac82b26f9947f0d1c2322"}
Oct 30 11:50:25.193561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.192921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:25.809673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.809642    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:25.925686 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.925581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca\") pod \"8183a93a-076c-4dc1-9daf-30ee1c49733b\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") "
Oct 30 11:50:25.925686 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.925667    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca\") pod \"8183a93a-076c-4dc1-9daf-30ee1c49733b\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") "
Oct 30 11:50:25.925924 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.925734    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr\" (UniqueName: \"kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr\") pod \"8183a93a-076c-4dc1-9daf-30ee1c49733b\" (UID: \"8183a93a-076c-4dc1-9daf-30ee1c49733b\") "
Oct 30 11:50:25.926087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.926058    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca" pod "8183a93a-076c-4dc1-9daf-30ee1c49733b" (UID: "8183a93a-076c-4dc1-9daf-30ee1c49733b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:50:25.926231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.926216    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:25.928046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.928020    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr" pod "8183a93a-076c-4dc1-9daf-30ee1c49733b" (UID: "8183a93a-076c-4dc1-9daf-30ee1c49733b"). InnerVolumeSpecName "kube-api-access-lnclr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:50:25.928174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:25.928076    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca" pod "8183a93a-076c-4dc1-9daf-30ee1c49733b" (UID: "8183a93a-076c-4dc1-9daf-30ee1c49733b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:50:26.027956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:26.027915    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8183a93a-076c-4dc1-9daf-30ee1c49733b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:26.027956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:26.027957    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnclr\" (UniqueName: \"kubernetes.io/projected/8183a93a-076c-4dc1-9daf-30ee1c49733b-kube-api-access-lnclr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:26.645158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:26.645070    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss" event={"ID":"8183a93a-076c-4dc1-9daf-30ee1c49733b","Type":"ContainerDied","Data":"888403246df7aaa8c224156024e042462315aa73be27bc82c34d3e21158dde3a"}
Oct 30 11:50:26.645158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:26.645112    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"
Oct 30 11:50:26.645359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:26.645118    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888403246df7aaa8c224156024e042462315aa73be27bc82c34d3e21158dde3a"
Oct 30 11:50:26.949672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:26.949643    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:28.683097 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:28.683052    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:30.093482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:30.091653    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:30.844539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:30.843487    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.232840    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7374-21cb27"]
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.233779    5060 cpu_manager.go:414] "containerMap: removing container" podUID="19a5fe67-da11-4387-a313-e9f0391328f4" containerName="revoke-keys"
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.233802    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a5fe67-da11-4387-a313-e9f0391328f4" containerName="revoke-keys"
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.233839    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8183a93a-076c-4dc1-9daf-30ee1c49733b" containerName="revoke-keys"
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.233848    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8183a93a-076c-4dc1-9daf-30ee1c49733b" containerName="revoke-keys"
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.234051    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a5fe67-da11-4387-a313-e9f0391328f4" containerName="revoke-keys"
Oct 30 11:50:31.235684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.234069    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8183a93a-076c-4dc1-9daf-30ee1c49733b" containerName="revoke-keys"
Oct 30 11:50:31.242179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.240591    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:31.258710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.257894    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\""
Oct 30 11:50:31.270738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.270668    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7374-21cb27"]
Oct 30 11:50:31.367496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.342406    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7vj\" (UniqueName: \"kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj\") pod \"test-curl-7374-21cb27\" (UID: \"8712a523-c829-4117-8f57-808581d6ef0b\") " pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:31.469879 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.469842    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7vj\" (UniqueName: \"kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj\") pod \"test-curl-7374-21cb27\" (UID: \"8712a523-c829-4117-8f57-808581d6ef0b\") " pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:31.494494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.494398    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7vj\" (UniqueName: \"kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj\") pod \"test-curl-7374-21cb27\" (UID: \"8712a523-c829-4117-8f57-808581d6ef0b\") " pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:31.563281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.563243    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:31.572312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.572284    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:50:31.882836 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:31.882800    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8712a523_c829_4117_8f57_808581d6ef0b.slice/crio-880046627f509310042721eed344d9a9d4a47429059a748ac21cb04a460c2425 WatchSource:0}: Error finding container 880046627f509310042721eed344d9a9d4a47429059a748ac21cb04a460c2425: Status 404 returned error can't find the container with id 880046627f509310042721eed344d9a9d4a47429059a748ac21cb04a460c2425
Oct 30 11:50:31.884539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.884515    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:50:31.884649 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:31.884594    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7374-21cb27"]
Oct 30 11:50:32.733677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:32.733647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:32.758605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:32.758564    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7374-21cb27" event={"ID":"8712a523-c829-4117-8f57-808581d6ef0b","Type":"ContainerStarted","Data":"880046627f509310042721eed344d9a9d4a47429059a748ac21cb04a460c2425"}
Oct 30 11:50:32.892439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:32.892397    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:34.756713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:34.756647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:35.787058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:35.787025    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7374-21cb27" event={"ID":"8712a523-c829-4117-8f57-808581d6ef0b","Type":"ContainerStarted","Data":"18a74094e739c139c5e5832852c66c771418ca4bc8a24a4a2a4d64936f22dab6"}
Oct 30 11:50:35.827030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:35.826974    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7374-21cb27" podStartSLOduration=1.930676682 podStartE2EDuration="4.826960103s" totalImagesPullingTime="2.896283421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:35.825052284 +0000 UTC m=+1239.619936775" watchObservedRunningTime="2026-10-30 11:50:35.826960103 +0000 UTC m=+1239.621844597"
Oct 30 11:50:35.985199 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:35.985157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:36.357608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.357569    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:36.458229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.458192    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"]
Oct 30 11:50:36.462719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.462693    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.484701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.484669    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-93525b-openshift-default-dockercfg-pwzb9\""
Oct 30 11:50:36.508007 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.507970    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"]
Oct 30 11:50:36.563437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.563399    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:36.568621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.568589    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.568776 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.568662    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.568776 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.568696    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.568870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.568849    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.568907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.568882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.569008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.568997    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.569066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.569054    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.569118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.569108    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wb4d\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.569176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.569162    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670497    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670636    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wb4d\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670686    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670736    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670784    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670810    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670922    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.670950    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.674601    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.674911    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.674923    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.675147    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.675567    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.676535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.675837    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.706489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.705079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wb4d\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.709429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.708302    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.715397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.710896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo\") pod \"e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") " pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.715397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.715277    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:36.790546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.788750    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:36.870476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:36.865925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:37.194510 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:37.194479    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbac8954_3288_4d1d_95a6_a29b04e45490.slice/crio-aad9e10eee549a73ceda703329e91dac53a8eae116e45f50dfb4e7f213643a38 WatchSource:0}: Error finding container aad9e10eee549a73ceda703329e91dac53a8eae116e45f50dfb4e7f213643a38: Status 404 returned error can't find the container with id aad9e10eee549a73ceda703329e91dac53a8eae116e45f50dfb4e7f213643a38
Oct 30 11:50:37.194510 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.194487    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"]
Oct 30 11:50:37.196901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.196869    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:50:37.197030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.196950    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:50:37.197030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.196988    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:50:37.823604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.823568    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" event={"ID":"dbac8954-3288-4d1d-95a6-a29b04e45490","Type":"ContainerStarted","Data":"60637fbc17a8c99ec1681760cec5529a22eb8b3b01d657abc74689c19948aca3"}
Oct 30 11:50:37.824109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.823632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" event={"ID":"dbac8954-3288-4d1d-95a6-a29b04e45490","Type":"ContainerStarted","Data":"aad9e10eee549a73ceda703329e91dac53a8eae116e45f50dfb4e7f213643a38"}
Oct 30 11:50:37.898661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:37.898588    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podStartSLOduration=1.898565784 podStartE2EDuration="1.898565784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:37.881497256 +0000 UTC m=+1241.676381750" watchObservedRunningTime="2026-10-30 11:50:37.898565784 +0000 UTC m=+1241.693450279"
Oct 30 11:50:38.435661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:38.435620    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:38.789656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:38.789620    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:38.796555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:38.796526    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:38.832104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:38.832072    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:38.833089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:38.833070    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:50:40.199496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:40.195526    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:41.958547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:41.958503    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:43.837479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:43.837427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:46.259868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:46.259816    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:46.581773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:46.577109    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:46.613478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:46.593142    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:46.620609 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:46.620557    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:46.825511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:46.816638    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:46.825511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:46.824365    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:47.059478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.058607    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:50:47.580478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.579541    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:47.580478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.580305    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:50:47.638875 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.626731    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:50:47.638875 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.627207    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:50:47.692646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.692596    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:47.752427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:47.752395    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:50:48.121258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.121214    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 11:50:48.264837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.264775    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:48.344479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.276980    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:50:48.494961 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.494922    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:48.531487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.516500    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:50:48.531487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.516608    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:50:48.590476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.581218    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:48.629117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.609238    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:50:48.756094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.749389    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:50:48.968108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:48.968048    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:49.084689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.083830    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7374-21cb27"]
Oct 30 11:50:49.084689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.084044    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7374-21cb27" podUID="8712a523-c829-4117-8f57-808581d6ef0b" containerName="test-curl-7374-21cb27" containerID="cri-o://18a74094e739c139c5e5832852c66c771418ca4bc8a24a4a2a4d64936f22dab6" gracePeriod=2
Oct 30 11:50:49.094963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.094880    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7374-21cb27"]
Oct 30 11:50:49.098174 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:50:49.098132    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7374-21cb27\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="8712a523-c829-4117-8f57-808581d6ef0b" pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:49.790350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.790308    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"]
Oct 30 11:50:49.791017 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.790998    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8712a523-c829-4117-8f57-808581d6ef0b" containerName="test-curl-7374-21cb27"
Oct 30 11:50:49.791104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.791017    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8712a523-c829-4117-8f57-808581d6ef0b" containerName="test-curl-7374-21cb27"
Oct 30 11:50:49.791194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.791182    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8712a523-c829-4117-8f57-808581d6ef0b" containerName="test-curl-7374-21cb27"
Oct 30 11:50:49.795200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.795176    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.815560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.810651    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:50:49.815560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.813377    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"]
Oct 30 11:50:49.822789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.822759    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:50:49.823894 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.823102    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:50:49.824050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.823378    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:50:49.861346 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:50:49.861301    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7374-21cb27\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="8712a523-c829-4117-8f57-808581d6ef0b" pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:49.864171 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:50:49.864135    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:50:49.872073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.872042    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.872218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.872189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gth5w\" (UniqueName: \"kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.872317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.872303    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.973359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.973321    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gth5w\" (UniqueName: \"kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.973579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.973426    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.973579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.973567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.974221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.974195    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.976054 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.976030    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:49.998319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:49.998281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gth5w\" (UniqueName: \"kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:50.115198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:50.115094    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:50.429774 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:50:50.429743    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee3a8f1_261b_4b70_88e3_39ec596bf7ee.slice/crio-63a0c045f8940fac138e64b119a7991e902c7672bfcf4bc2bcd3337da7bd0c9e WatchSource:0}: Error finding container 63a0c045f8940fac138e64b119a7991e902c7672bfcf4bc2bcd3337da7bd0c9e: Status 404 returned error can't find the container with id 63a0c045f8940fac138e64b119a7991e902c7672bfcf4bc2bcd3337da7bd0c9e
Oct 30 11:50:50.431182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:50.431157    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"]
Oct 30 11:50:50.615085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:50.615044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:50.783350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:50.783310    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:51.341586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.341499    5060 generic.go:356] "Generic (PLEG): container finished" podID="8712a523-c829-4117-8f57-808581d6ef0b" containerID="18a74094e739c139c5e5832852c66c771418ca4bc8a24a4a2a4d64936f22dab6" exitCode=137
Oct 30 11:50:51.341586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.341550    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="880046627f509310042721eed344d9a9d4a47429059a748ac21cb04a460c2425"
Oct 30 11:50:51.343370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.343343    5060 generic.go:356] "Generic (PLEG): container finished" podID="eee3a8f1-261b-4b70-88e3-39ec596bf7ee" containerID="b0c3774e0ca63bc0801c82b62c2df86ead568893ccf810235ea7e580034a32e4" exitCode=0
Oct 30 11:50:51.343527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.343425    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9" event={"ID":"eee3a8f1-261b-4b70-88e3-39ec596bf7ee","Type":"ContainerDied","Data":"b0c3774e0ca63bc0801c82b62c2df86ead568893ccf810235ea7e580034a32e4"}
Oct 30 11:50:51.343527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.343481    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9" event={"ID":"eee3a8f1-261b-4b70-88e3-39ec596bf7ee","Type":"ContainerStarted","Data":"63a0c045f8940fac138e64b119a7991e902c7672bfcf4bc2bcd3337da7bd0c9e"}
Oct 30 11:50:51.400873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.400847    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:51.500839 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.500797    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj\" (UniqueName: \"kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj\") pod \"8712a523-c829-4117-8f57-808581d6ef0b\" (UID: \"8712a523-c829-4117-8f57-808581d6ef0b\") "
Oct 30 11:50:51.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.503118    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj" pod "8712a523-c829-4117-8f57-808581d6ef0b" (UID: "8712a523-c829-4117-8f57-808581d6ef0b"). InnerVolumeSpecName "kube-api-access-2s7vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:50:51.602851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:51.602750    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s7vj\" (UniqueName: \"kubernetes.io/projected/8712a523-c829-4117-8f57-808581d6ef0b-kube-api-access-2s7vj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:52.303481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.303428    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:52.336054 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.336023    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8712a523-c829-4117-8f57-808581d6ef0b" path="/var/lib/kubelet/pods/8712a523-c829-4117-8f57-808581d6ef0b/volumes"
Oct 30 11:50:52.350550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.350525    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-21cb27"
Oct 30 11:50:52.474264 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.474229    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:52.742631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.742600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:52.809726 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.809693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:52.853159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.853126    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w\" (UniqueName: \"kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w\") pod \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") "
Oct 30 11:50:52.853374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.853269    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca\") pod \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") "
Oct 30 11:50:52.853374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.853355    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca\") pod \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\" (UID: \"eee3a8f1-261b-4b70-88e3-39ec596bf7ee\") "
Oct 30 11:50:52.853683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.853656    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca" pod "eee3a8f1-261b-4b70-88e3-39ec596bf7ee" (UID: "eee3a8f1-261b-4b70-88e3-39ec596bf7ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:50:52.854348 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.854330    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:52.855561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.855539    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w" pod "eee3a8f1-261b-4b70-88e3-39ec596bf7ee" (UID: "eee3a8f1-261b-4b70-88e3-39ec596bf7ee"). InnerVolumeSpecName "kube-api-access-gth5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:50:52.855641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.855558    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca" pod "eee3a8f1-261b-4b70-88e3-39ec596bf7ee" (UID: "eee3a8f1-261b-4b70-88e3-39ec596bf7ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:50:52.956213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.956177    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:52.956213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:52.956210    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gth5w\" (UniqueName: \"kubernetes.io/projected/eee3a8f1-261b-4b70-88e3-39ec596bf7ee-kube-api-access-gth5w\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:50:53.359605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:53.359505    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"
Oct 30 11:50:53.359605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:53.359515    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9" event={"ID":"eee3a8f1-261b-4b70-88e3-39ec596bf7ee","Type":"ContainerDied","Data":"63a0c045f8940fac138e64b119a7991e902c7672bfcf4bc2bcd3337da7bd0c9e"}
Oct 30 11:50:53.359605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:53.359575    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a0c045f8940fac138e64b119a7991e902c7672bfcf4bc2bcd3337da7bd0c9e"
Oct 30 11:50:54.861668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:54.861631    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:56.687253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:56.687216    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:50:58.457156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:50:58.457110    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:00.390491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:00.388275    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:00.538013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:00.537977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:02.366445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:02.366403    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:02.927796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:02.927743    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:04.249127 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:04.249086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:05.706783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.706740    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"]
Oct 30 11:51:05.707580 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.707549    5060 cpu_manager.go:414] "containerMap: removing container" podUID="eee3a8f1-261b-4b70-88e3-39ec596bf7ee" containerName="revoke-keys"
Oct 30 11:51:05.707580 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.707573    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee3a8f1-261b-4b70-88e3-39ec596bf7ee" containerName="revoke-keys"
Oct 30 11:51:05.707789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.707694    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee3a8f1-261b-4b70-88e3-39ec596bf7ee" containerName="revoke-keys"
Oct 30 11:51:05.711384 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.711364    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.727517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.727486    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:51:05.729092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.729067    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"]
Oct 30 11:51:05.737117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.737089    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:51:05.737340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.737093    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:51:05.737340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.737121    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:51:05.842833 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.842791    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.843019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.842973    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.843224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.843205    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zts\" (UniqueName: \"kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.944933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.944899    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.945149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.944972    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.945149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.945084    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zts\" (UniqueName: \"kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.945632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.945605    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.947710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.947683    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:05.969436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:05.969358    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zts\" (UniqueName: \"kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts\") pod \"maas-api-revoke-sub-97842b20-pvc8j\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:06.027567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:06.027532    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:06.117901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:06.117610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:06.264437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:06.264351    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:06.341239 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:51:06.341209    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda430c672_5d59_431d_baed_bf67c11ea117.slice/crio-f98b0207c0ae66d649d7d2d597d29afa086a341a903e684ec292ed105ca9be1b WatchSource:0}: Error finding container f98b0207c0ae66d649d7d2d597d29afa086a341a903e684ec292ed105ca9be1b: Status 404 returned error can't find the container with id f98b0207c0ae66d649d7d2d597d29afa086a341a903e684ec292ed105ca9be1b
Oct 30 11:51:06.344257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:06.344144    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"]
Oct 30 11:51:06.487927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:06.487890    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j" event={"ID":"a430c672-5d59-431d-baed-bf67c11ea117","Type":"ContainerStarted","Data":"f98b0207c0ae66d649d7d2d597d29afa086a341a903e684ec292ed105ca9be1b"}
Oct 30 11:51:07.497646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:07.497608    5060 generic.go:356] "Generic (PLEG): container finished" podID="a430c672-5d59-431d-baed-bf67c11ea117" containerID="694c32e6ae373675b8013fd192b6db825c5b7c71adc8610c2e1fd2e4a2a6e690" exitCode=0
Oct 30 11:51:07.498113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:07.497703    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j" event={"ID":"a430c672-5d59-431d-baed-bf67c11ea117","Type":"ContainerDied","Data":"694c32e6ae373675b8013fd192b6db825c5b7c71adc8610c2e1fd2e4a2a6e690"}
Oct 30 11:51:08.065570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.065516    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:08.686763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.686720    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:08.806298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.806214    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca\") pod \"a430c672-5d59-431d-baed-bf67c11ea117\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") "
Oct 30 11:51:08.806298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.806268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts\" (UniqueName: \"kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts\") pod \"a430c672-5d59-431d-baed-bf67c11ea117\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") "
Oct 30 11:51:08.806654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.806408    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca\") pod \"a430c672-5d59-431d-baed-bf67c11ea117\" (UID: \"a430c672-5d59-431d-baed-bf67c11ea117\") "
Oct 30 11:51:08.806757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.806668    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca" pod "a430c672-5d59-431d-baed-bf67c11ea117" (UID: "a430c672-5d59-431d-baed-bf67c11ea117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:51:08.806825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.806815    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:08.808806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.808774    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts" pod "a430c672-5d59-431d-baed-bf67c11ea117" (UID: "a430c672-5d59-431d-baed-bf67c11ea117"). InnerVolumeSpecName "kube-api-access-n7zts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:51:08.808948 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.808894    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca" pod "a430c672-5d59-431d-baed-bf67c11ea117" (UID: "a430c672-5d59-431d-baed-bf67c11ea117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:51:08.908177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.908133    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a430c672-5d59-431d-baed-bf67c11ea117-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:08.908177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:08.908170    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zts\" (UniqueName: \"kubernetes.io/projected/a430c672-5d59-431d-baed-bf67c11ea117-kube-api-access-n7zts\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:09.515763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:09.515732    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"
Oct 30 11:51:09.515980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:09.515767    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j" event={"ID":"a430c672-5d59-431d-baed-bf67c11ea117","Type":"ContainerDied","Data":"f98b0207c0ae66d649d7d2d597d29afa086a341a903e684ec292ed105ca9be1b"}
Oct 30 11:51:09.515980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:09.515821    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98b0207c0ae66d649d7d2d597d29afa086a341a903e684ec292ed105ca9be1b"
Oct 30 11:51:09.852966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:09.852867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:11.653770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:11.653711    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:13.444628 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:13.444593    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:15.267192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.267155    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:15.899936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.899893    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"]
Oct 30 11:51:15.900867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.900846    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a430c672-5d59-431d-baed-bf67c11ea117" containerName="revoke-keys"
Oct 30 11:51:15.900969 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.900870    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a430c672-5d59-431d-baed-bf67c11ea117" containerName="revoke-keys"
Oct 30 11:51:15.901062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.901051    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a430c672-5d59-431d-baed-bf67c11ea117" containerName="revoke-keys"
Oct 30 11:51:15.905298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.905273    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:15.909375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.909341    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:15.909538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.909400    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:15.909538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.909493    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9sv7\" (UniqueName: \"kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:15.922773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.922735    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"]
Oct 30 11:51:15.928230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.928194    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:51:15.928230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.928194    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:51:15.928507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.928207    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:51:15.928507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:15.928277    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:51:16.010933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.010889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.011145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.010979    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.011145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.011014    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9sv7\" (UniqueName: \"kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.011774 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.011677    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.013526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.013506    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.035424 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.035385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9sv7\" (UniqueName: \"kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7\") pod \"maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.224160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.224121    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:16.539088 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:51:16.539056    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ea1d8a_a4f2_4e7a_81ba_f413c81f781d.slice/crio-967caf121ab59f7f5721585ad969c3cff121c3dc9e36de3f5ed50355fc976e69 WatchSource:0}: Error finding container 967caf121ab59f7f5721585ad969c3cff121c3dc9e36de3f5ed50355fc976e69: Status 404 returned error can't find the container with id 967caf121ab59f7f5721585ad969c3cff121c3dc9e36de3f5ed50355fc976e69
Oct 30 11:51:16.540945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.540922    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"]
Oct 30 11:51:16.588696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:16.588644    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8" event={"ID":"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d","Type":"ContainerStarted","Data":"967caf121ab59f7f5721585ad969c3cff121c3dc9e36de3f5ed50355fc976e69"}
Oct 30 11:51:17.181560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:17.169247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:17.598359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:17.598321    5060 generic.go:356] "Generic (PLEG): container finished" podID="c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" containerID="934355da24bf16dfca8152b50361c5a68db437596d8bee2aeb775f304feb9567" exitCode=0
Oct 30 11:51:17.598903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:17.598369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8" event={"ID":"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d","Type":"ContainerDied","Data":"934355da24bf16dfca8152b50361c5a68db437596d8bee2aeb775f304feb9567"}
Oct 30 11:51:18.901520 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.901430    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:18.963696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.963656    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca\") pod \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") "
Oct 30 11:51:18.963902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.963797    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca\") pod \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") "
Oct 30 11:51:18.963902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.963856    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7\" (UniqueName: \"kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7\") pod \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\" (UID: \"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d\") "
Oct 30 11:51:18.964292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.964257    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca" pod "c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" (UID: "c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:51:18.964545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.964525    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:18.966590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.966559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca" pod "c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" (UID: "c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:51:18.966714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:18.966695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7" pod "c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" (UID: "c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d"). InnerVolumeSpecName "kube-api-access-r9sv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:51:19.066118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:19.066081    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:19.066118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:19.066116    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9sv7\" (UniqueName: \"kubernetes.io/projected/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d-kube-api-access-r9sv7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:19.292408 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:19.292370    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:19.650351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:19.650265    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8" event={"ID":"c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d","Type":"ContainerDied","Data":"967caf121ab59f7f5721585ad969c3cff121c3dc9e36de3f5ed50355fc976e69"}
Oct 30 11:51:19.650351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:19.650313    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="967caf121ab59f7f5721585ad969c3cff121c3dc9e36de3f5ed50355fc976e69"
Oct 30 11:51:19.650351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:19.650272    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"
Oct 30 11:51:19.859295 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:51:19.859259    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:51:21.208479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:21.204527    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:23.198477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:23.194108    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:25.216870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:25.216827    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:25.675900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:25.674145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:26.427810 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.427145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:26.570838 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.570803    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:26.890274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.890237    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"]
Oct 30 11:51:26.891276 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.891251    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" containerName="revoke-keys"
Oct 30 11:51:26.891276 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.891276    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" containerName="revoke-keys"
Oct 30 11:51:26.891429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.891383    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" containerName="revoke-keys"
Oct 30 11:51:26.895350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.895325    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:26.928633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.927927    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b7bf55-openshift-default-dockercfg-hrxhg\""
Oct 30 11:51:26.950621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:26.949886    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"]
Oct 30 11:51:27.002477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.001046    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:27.030488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.030417    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.030656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.030566    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.030656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.030635    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.030803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.030779    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.030932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.030911    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.031008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.030990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.031110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.031092    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.031236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.031206    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.031371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.031251    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133261    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133400    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133442    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133490    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133569    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133619    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133686    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.133762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133755    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.134028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.133964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.134125 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.134103    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.134180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.134145    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.134826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.134563    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.134826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.134789    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.136684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.136660    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.136792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.136729    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.155320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.154988    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:27.157995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.157969    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.158264 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.158238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq\") pod \"e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") " pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.230605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.230527    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:27.862201 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.862107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:27.893486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.893345    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"]
Oct 30 11:51:27.894986 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:51:27.894935    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe30b482_dd34_4a19_a5b0_4813dc758914.slice/crio-e8bd6e24d7810b7a36fc023260d6daf221e8fb0c4196ea222dfab5199d3e4001 WatchSource:0}: Error finding container e8bd6e24d7810b7a36fc023260d6daf221e8fb0c4196ea222dfab5199d3e4001: Status 404 returned error can't find the container with id e8bd6e24d7810b7a36fc023260d6daf221e8fb0c4196ea222dfab5199d3e4001
Oct 30 11:51:27.897558 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.897523    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:51:27.897669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.897603    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:51:27.897669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:27.897648    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:51:28.056265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.056116    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:28.073485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.070565    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"]
Oct 30 11:51:28.079407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.079370    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-54gf9"]
Oct 30 11:51:28.212094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.212051    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:28.338683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.338497    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee3a8f1-261b-4b70-88e3-39ec596bf7ee" path="/var/lib/kubelet/pods/eee3a8f1-261b-4b70-88e3-39ec596bf7ee/volumes"
Oct 30 11:51:28.794816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.794766    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" event={"ID":"be30b482-dd34-4a19-a5b0-4813dc758914","Type":"ContainerStarted","Data":"c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"}
Oct 30 11:51:28.795029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:28.794840    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" event={"ID":"be30b482-dd34-4a19-a5b0-4813dc758914","Type":"ContainerStarted","Data":"e8bd6e24d7810b7a36fc023260d6daf221e8fb0c4196ea222dfab5199d3e4001"}
Oct 30 11:51:29.231713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.231681    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:29.238521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.238491    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:29.287292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.287237    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podStartSLOduration=3.287221249 podStartE2EDuration="3.287221249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:51:28.854221963 +0000 UTC m=+1292.649106458" watchObservedRunningTime="2026-10-30 11:51:29.287221249 +0000 UTC m=+1293.082105743"
Oct 30 11:51:29.805757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.805726    5060 generic.go:356] "Generic (PLEG): container finished" podID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerID="60637fbc17a8c99ec1681760cec5529a22eb8b3b01d657abc74689c19948aca3" exitCode=0
Oct 30 11:51:29.806006 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.805791    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" event={"ID":"dbac8954-3288-4d1d-95a6-a29b04e45490","Type":"ContainerDied","Data":"60637fbc17a8c99ec1681760cec5529a22eb8b3b01d657abc74689c19948aca3"}
Oct 30 11:51:29.806220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.806201    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:29.806322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.806207    5060 scope.go:122] "RemoveContainer" containerID="60637fbc17a8c99ec1681760cec5529a22eb8b3b01d657abc74689c19948aca3"
Oct 30 11:51:29.807506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.807446    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:51:29.808078 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.808048    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:51:29.808154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.808138    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:51:29.808202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:29.808188    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:51:30.059027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:30.058925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:30.818557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:30.818520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" event={"ID":"dbac8954-3288-4d1d-95a6-a29b04e45490","Type":"ContainerStarted","Data":"f4219dd77f37bb7ca0e33681d78431542774fe981e7290536a733de220bdc629"}
Oct 30 11:51:31.116181 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.116080    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:31.287008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.286976    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:31.442655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.442573    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:31.595994 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.595960    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:31.799476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.799248    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:51:31.799476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.799395    5060 patch_prober.go:29] interesting pod/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 11:51:31.799476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:31.799472    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 11:51:32.011267 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:32.011045    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:32.795904 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:32.795861    5060 patch_prober.go:29] interesting pod/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 11:51:32.796280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:32.796245    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 11:51:33.107007 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:33.106901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:33.255158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:33.255116    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:33.799912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:33.799865    5060 patch_prober.go:29] interesting pod/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 11:51:33.800140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:33.799935    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 11:51:34.205479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:34.199977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:34.794478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:34.792865    5060 patch_prober.go:29] interesting pod/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 11:51:34.794478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:34.792927    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 11:51:35.797890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:35.797853    5060 patch_prober.go:29] interesting pod/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 11:51:35.798523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:35.798490    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 11:51:35.951279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:35.951239    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:36.795320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:36.795237    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:51:36.795700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:36.795674    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:51:36.796588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:36.796561    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:51:37.858742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:37.858523    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:39.631082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:39.631048    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:41.365978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:41.365948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:42.888522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:42.888482    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:43.486650 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:43.486609    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:43.647572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:43.647538    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"]
Oct 30 11:51:43.651327 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:43.651291    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:43.661367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:43.661327    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71af6f-jl2g8"]
Oct 30 11:51:43.800391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:43.800295    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:44.237355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:44.237319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:44.335608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:44.335575    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d" path="/var/lib/kubelet/pods/c6ea1d8a-a4f2-4e7a-81ba-f413c81f781d/volumes"
Oct 30 11:51:45.536982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:45.536949    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:45.698560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:45.698519    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:45.855080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:45.854980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:47.657529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:47.657440    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:49.150811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:49.150722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:49.305280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:49.305247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:49.606105 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:49.606067    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:49.860094 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:51:49.860009    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:51:51.385162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:51.385124    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:51.534290 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:51.534248    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:51.694782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:51.693238    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:51.840694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:51.840652    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:52.276851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.276805    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"]
Oct 30 11:51:52.282620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.282583    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.299200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.299168    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"]
Oct 30 11:51:52.306597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.306568    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:51:52.307504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.307431    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:51:52.311577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.311539    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.311752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.311603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.311752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.311675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qc7s\" (UniqueName: \"kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.342485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.342438    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:51:52.342675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.342518    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:51:52.413662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.413611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.413662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.413663    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.414146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.413704    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qc7s\" (UniqueName: \"kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.414506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.414480    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.416358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.416334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.421885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.421842    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:52.449739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.449707    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qc7s\" (UniqueName: \"kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s\") pod \"maas-api-revoke-keys-e2e-api-b-58826b-698kh\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.600853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.600755    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:52.922812 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:51:52.922784    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd491c8e3_da28_451c_8276_a0cbef2d0c42.slice/crio-ba60bc9543137fec437595c5a741a6c79eeb1a273cdc1aa95d546041e1221aeb WatchSource:0}: Error finding container ba60bc9543137fec437595c5a741a6c79eeb1a273cdc1aa95d546041e1221aeb: Status 404 returned error can't find the container with id ba60bc9543137fec437595c5a741a6c79eeb1a273cdc1aa95d546041e1221aeb
Oct 30 11:51:52.927168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:52.927041    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"]
Oct 30 11:51:53.017176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:53.017140    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh" event={"ID":"d491c8e3-da28-451c-8276-a0cbef2d0c42","Type":"ContainerStarted","Data":"ba60bc9543137fec437595c5a741a6c79eeb1a273cdc1aa95d546041e1221aeb"}
Oct 30 11:51:53.919553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:53.919519    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:54.027993 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:54.027888    5060 generic.go:356] "Generic (PLEG): container finished" podID="d491c8e3-da28-451c-8276-a0cbef2d0c42" containerID="7768f10a971db5e142345388e08b41945ae14555f10f625c0d601b49c1860481" exitCode=0
Oct 30 11:51:54.028189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:54.027986    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh" event={"ID":"d491c8e3-da28-451c-8276-a0cbef2d0c42","Type":"ContainerDied","Data":"7768f10a971db5e142345388e08b41945ae14555f10f625c0d601b49c1860481"}
Oct 30 11:51:55.213958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.213928    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:55.252186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.252152    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca\") pod \"d491c8e3-da28-451c-8276-a0cbef2d0c42\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") "
Oct 30 11:51:55.252365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.252334    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca\") pod \"d491c8e3-da28-451c-8276-a0cbef2d0c42\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") "
Oct 30 11:51:55.252413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.252371    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s\" (UniqueName: \"kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s\") pod \"d491c8e3-da28-451c-8276-a0cbef2d0c42\" (UID: \"d491c8e3-da28-451c-8276-a0cbef2d0c42\") "
Oct 30 11:51:55.252722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.252693    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca" pod "d491c8e3-da28-451c-8276-a0cbef2d0c42" (UID: "d491c8e3-da28-451c-8276-a0cbef2d0c42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:51:55.253103 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.253067    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:55.255107 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.255078    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca" pod "d491c8e3-da28-451c-8276-a0cbef2d0c42" (UID: "d491c8e3-da28-451c-8276-a0cbef2d0c42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:51:55.255395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.255370    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s" pod "d491c8e3-da28-451c-8276-a0cbef2d0c42" (UID: "d491c8e3-da28-451c-8276-a0cbef2d0c42"). InnerVolumeSpecName "kube-api-access-2qc7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:51:55.354798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.354686    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qc7s\" (UniqueName: \"kubernetes.io/projected/d491c8e3-da28-451c-8276-a0cbef2d0c42-kube-api-access-2qc7s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:55.354798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.354726    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d491c8e3-da28-451c-8276-a0cbef2d0c42-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:51:55.681731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:55.681601    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:56.047770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:56.047736    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"
Oct 30 11:51:56.047949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:56.047769    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh" event={"ID":"d491c8e3-da28-451c-8276-a0cbef2d0c42","Type":"ContainerDied","Data":"ba60bc9543137fec437595c5a741a6c79eeb1a273cdc1aa95d546041e1221aeb"}
Oct 30 11:51:56.047949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:56.047818    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba60bc9543137fec437595c5a741a6c79eeb1a273cdc1aa95d546041e1221aeb"
Oct 30 11:51:56.715395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:56.715359    5060 scope.go:122] "RemoveContainer" containerID="901a2414d11166f01e9d35dfe36d42891911e4e9b873f189e5969cfcee97d127"
Oct 30 11:51:57.507134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:57.507099    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:58.275103 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:58.275067    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:59.319913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:59.319848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:51:59.979522 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:51:59.979443    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:00.129168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:00.129128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:00.562806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:00.562771    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:00.819948 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:00.819838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:00.989062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:00.989011    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:01.515663 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:01.515621    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:01.700444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:01.700406    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:02.055575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:02.055540    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:02.335539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:02.332350    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:02.770145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:02.770100    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"]
Oct 30 11:52:02.965748 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:02.965701    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:03.156898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:03.156783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:03.752629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:03.752590    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:05.868413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:05.867145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:06.095478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:06.095002    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:06.264932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:06.264901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:06.876352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:06.876316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:07.798588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:07.798531    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6" podUID="d19047b8-ff68-4543-8a2a-d3abf4836a97" containerName="payload-processing" containerID="cri-o://172427c6cabcc9157665ad7a8becb5276ca50649c488ab7d76bd8ccbe01c2f70" gracePeriod=25
Oct 30 11:52:07.912483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:07.911420    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:08.062231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.062105    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"]
Oct 30 11:52:08.063668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.063644    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d491c8e3-da28-451c-8276-a0cbef2d0c42" containerName="revoke-keys"
Oct 30 11:52:08.063895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.063878    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d491c8e3-da28-451c-8276-a0cbef2d0c42" containerName="revoke-keys"
Oct 30 11:52:08.064248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.064230    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d491c8e3-da28-451c-8276-a0cbef2d0c42" containerName="revoke-keys"
Oct 30 11:52:08.070179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.069705    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.100013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.097344    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-caa7a0\""
Oct 30 11:52:08.122657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.114005    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-caa7a0-dockercfg-rxqsc\""
Oct 30 11:52:08.122657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.115137    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:08.139385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.139358    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"]
Oct 30 11:52:08.323749 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.281054    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"]
Oct 30 11:52:08.367173 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.342337    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:08.431551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.412859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.431551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.413201    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrdw\" (UniqueName: \"kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.473471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.473419    5060 generic.go:356] "Generic (PLEG): container finished" podID="d19047b8-ff68-4543-8a2a-d3abf4836a97" containerID="172427c6cabcc9157665ad7a8becb5276ca50649c488ab7d76bd8ccbe01c2f70" exitCode=0
Oct 30 11:52:08.475235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.473674    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.515307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.501904    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:52:08.515551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.502921    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:52:08.515662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.503684    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:52:08.515764 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.503760    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:52:08.515872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.515285    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.516206 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.516168    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrdw\" (UniqueName: \"kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.516992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.516969    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.518886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.518856    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6" event={"ID":"d19047b8-ff68-4543-8a2a-d3abf4836a97","Type":"ContainerDied","Data":"172427c6cabcc9157665ad7a8becb5276ca50649c488ab7d76bd8ccbe01c2f70"}
Oct 30 11:52:08.518989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.518910    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6" event={"ID":"d19047b8-ff68-4543-8a2a-d3abf4836a97","Type":"ContainerDied","Data":"f1567a7bf06d82363bf02a2ae3bdde037d5176d18a581a9660f5b2b72d7324ca"}
Oct 30 11:52:08.518989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.518926    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1567a7bf06d82363bf02a2ae3bdde037d5176d18a581a9660f5b2b72d7324ca"
Oct 30 11:52:08.518989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.518937    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"]
Oct 30 11:52:08.519939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.519922    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:52:08.540834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.540801    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrdw\" (UniqueName: \"kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.618946 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.618841    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume\") pod \"d19047b8-ff68-4543-8a2a-d3abf4836a97\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") "
Oct 30 11:52:08.622217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.619782    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf\" (UniqueName: \"kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf\") pod \"d19047b8-ff68-4543-8a2a-d3abf4836a97\" (UID: \"d19047b8-ff68-4543-8a2a-d3abf4836a97\") "
Oct 30 11:52:08.622217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.619336    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume" pod "d19047b8-ff68-4543-8a2a-d3abf4836a97" (UID: "d19047b8-ff68-4543-8a2a-d3abf4836a97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:52:08.622217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.620140    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.622217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.620264    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fklg\" (UniqueName: \"kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.622217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.621202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.622217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.621580    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19047b8-ff68-4543-8a2a-d3abf4836a97-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:52:08.624294 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.624266    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf" pod "d19047b8-ff68-4543-8a2a-d3abf4836a97" (UID: "d19047b8-ff68-4543-8a2a-d3abf4836a97"). InnerVolumeSpecName "kube-api-access-275kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:52:08.723120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.723081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.723326 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.723194    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.723326 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.723285    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fklg\" (UniqueName: \"kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.723549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.723529    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275kf\" (UniqueName: \"kubernetes.io/projected/d19047b8-ff68-4543-8a2a-d3abf4836a97-kube-api-access-275kf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:52:08.723821 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.723791    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.726683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.726658    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.744787 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.744753    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:08.748380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.748355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fklg\" (UniqueName: \"kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg\") pod \"maas-api-revoke-keys-e2e-api-a-93525b-mqlhj\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:08.828597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:08.828552    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:09.094131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.093895    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"]
Oct 30 11:52:09.095259 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:52:09.095219    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98568c1_21ca_4523_bb43_0fa8e383d5b7.slice/crio-71426d1e04e1945c31caa39dd3ea81bb63018d008078ff902a8fd38f68cf9bb5 WatchSource:0}: Error finding container 71426d1e04e1945c31caa39dd3ea81bb63018d008078ff902a8fd38f68cf9bb5: Status 404 returned error can't find the container with id 71426d1e04e1945c31caa39dd3ea81bb63018d008078ff902a8fd38f68cf9bb5
Oct 30 11:52:09.201566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.201537    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"]
Oct 30 11:52:09.203251 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:52:09.203211    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd370553c_a314_4f75_93dd_ede1c9ac5372.slice/crio-80489e8c6bf5d80ea0411a9c31d1c4d595c8d11c777d4579def4ee9889fc1d53 WatchSource:0}: Error finding container 80489e8c6bf5d80ea0411a9c31d1c4d595c8d11c777d4579def4ee9889fc1d53: Status 404 returned error can't find the container with id 80489e8c6bf5d80ea0411a9c31d1c4d595c8d11c777d4579def4ee9889fc1d53
Oct 30 11:52:09.489124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.489082    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj" event={"ID":"d370553c-a314-4f75-93dd-ede1c9ac5372","Type":"ContainerStarted","Data":"80489e8c6bf5d80ea0411a9c31d1c4d595c8d11c777d4579def4ee9889fc1d53"}
Oct 30 11:52:09.491057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.491026    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z" event={"ID":"e98568c1-21ca-4523-bb43-0fa8e383d5b7","Type":"ContainerStarted","Data":"0e01b8d9fcb1b7da7073aa190cf20dc972ba05996a4ca08cbb9d6d56a359d487"}
Oct 30 11:52:09.491218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.491081    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z" event={"ID":"e98568c1-21ca-4523-bb43-0fa8e383d5b7","Type":"ContainerStarted","Data":"71426d1e04e1945c31caa39dd3ea81bb63018d008078ff902a8fd38f68cf9bb5"}
Oct 30 11:52:09.491365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.491336    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:09.491539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.491516    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"
Oct 30 11:52:09.539937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.539880    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z" podStartSLOduration=1.53986086 podStartE2EDuration="1.53986086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:09.539698908 +0000 UTC m=+1333.334583406" watchObservedRunningTime="2026-10-30 11:52:09.53986086 +0000 UTC m=+1333.334745396"
Oct 30 11:52:09.586770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.586721    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"]
Oct 30 11:52:09.593553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:09.593513    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58826b-656cf46db6-5cln6"]
Oct 30 11:52:10.145809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.145776    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:10.337278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.337236    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19047b8-ff68-4543-8a2a-d3abf4836a97" path="/var/lib/kubelet/pods/d19047b8-ff68-4543-8a2a-d3abf4836a97/volumes"
Oct 30 11:52:10.504482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.504407    5060 generic.go:356] "Generic (PLEG): container finished" podID="d370553c-a314-4f75-93dd-ede1c9ac5372" containerID="b63378b3b21f79000c5e2976179af55c1e5633013b9a356349bd0926519cabd6" exitCode=0
Oct 30 11:52:10.504696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.504575    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj" event={"ID":"d370553c-a314-4f75-93dd-ede1c9ac5372","Type":"ContainerDied","Data":"b63378b3b21f79000c5e2976179af55c1e5633013b9a356349bd0926519cabd6"}
Oct 30 11:52:10.992700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.991185    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"]
Oct 30 11:52:10.992700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.992082    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d19047b8-ff68-4543-8a2a-d3abf4836a97" containerName="payload-processing"
Oct 30 11:52:10.992700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.992102    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19047b8-ff68-4543-8a2a-d3abf4836a97" containerName="payload-processing"
Oct 30 11:52:10.992700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.992271    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19047b8-ff68-4543-8a2a-d3abf4836a97" containerName="payload-processing"
Oct 30 11:52:10.996334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:10.996308    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.013193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.013167    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-84fed1-openshift-default-dockercfg-bxf5p\""
Oct 30 11:52:11.032316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.032285    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"]
Oct 30 11:52:11.088161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.088126    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172476    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172655    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172809    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172840    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172875    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8ng\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172908    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172950    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172974    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.175480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.172998    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.270533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.269591    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.276590    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.276750    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.276915    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.276953    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.276984    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8ng\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.277008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.277056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.277077    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.277097    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.281479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.281424    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.282156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.281720    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.282156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.281988    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.285476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.282283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.285476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.282978    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.306484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.306368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.322750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.321629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.327504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.323902    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.364478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.357560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8ng\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng\") pod \"e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") " pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.616997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.616726    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:11.699933 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.699538    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:11.803505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.803051    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca\") pod \"d370553c-a314-4f75-93dd-ede1c9ac5372\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") "
Oct 30 11:52:11.803505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.803106    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg\" (UniqueName: \"kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg\") pod \"d370553c-a314-4f75-93dd-ede1c9ac5372\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") "
Oct 30 11:52:11.803505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.803131    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca\") pod \"d370553c-a314-4f75-93dd-ede1c9ac5372\" (UID: \"d370553c-a314-4f75-93dd-ede1c9ac5372\") "
Oct 30 11:52:11.805046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.804702    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca" pod "d370553c-a314-4f75-93dd-ede1c9ac5372" (UID: "d370553c-a314-4f75-93dd-ede1c9ac5372"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:52:11.806584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.806552    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca" pod "d370553c-a314-4f75-93dd-ede1c9ac5372" (UID: "d370553c-a314-4f75-93dd-ede1c9ac5372"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:52:11.807674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.807646    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg" pod "d370553c-a314-4f75-93dd-ede1c9ac5372" (UID: "d370553c-a314-4f75-93dd-ede1c9ac5372"). InnerVolumeSpecName "kube-api-access-6fklg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:52:11.905547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.905442    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fklg\" (UniqueName: \"kubernetes.io/projected/d370553c-a314-4f75-93dd-ede1c9ac5372-kube-api-access-6fklg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:52:11.905547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.905502    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:52:11.905547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.905517    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d370553c-a314-4f75-93dd-ede1c9ac5372-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:52:11.988235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:11.988192    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"]
Oct 30 11:52:12.000345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.000302    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:52:12.000493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.000402    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:52:12.000493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.000442    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:52:12.054286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.054245    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:12.525660 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.525606    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" event={"ID":"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76","Type":"ContainerStarted","Data":"9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"}
Oct 30 11:52:12.526192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.525682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" event={"ID":"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76","Type":"ContainerStarted","Data":"4cc1be2c077b32c1b41f6440370902b9b9c1e28b6edd7d54ddac17e36ec8499f"}
Oct 30 11:52:12.527434 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.527401    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj" event={"ID":"d370553c-a314-4f75-93dd-ede1c9ac5372","Type":"ContainerDied","Data":"80489e8c6bf5d80ea0411a9c31d1c4d595c8d11c777d4579def4ee9889fc1d53"}
Oct 30 11:52:12.527564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.527446    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"
Oct 30 11:52:12.527564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.527501    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80489e8c6bf5d80ea0411a9c31d1c4d595c8d11c777d4579def4ee9889fc1d53"
Oct 30 11:52:12.582901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.582791    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podStartSLOduration=2.582771158 podStartE2EDuration="2.582771158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:12.579675778 +0000 UTC m=+1336.374560272" watchObservedRunningTime="2026-10-30 11:52:12.582771158 +0000 UTC m=+1336.377655655"
Oct 30 11:52:12.617486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:12.617416    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:13.623391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:13.623364    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:13.993435 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:13.993399    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:14.550554 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.550519    5060 generic.go:356] "Generic (PLEG): container finished" podID="be30b482-dd34-4a19-a5b0-4813dc758914" containerID="c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f" exitCode=0
Oct 30 11:52:14.550777 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.550591    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" event={"ID":"be30b482-dd34-4a19-a5b0-4813dc758914","Type":"ContainerDied","Data":"c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"}
Oct 30 11:52:14.551063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.551041    5060 scope.go:122] "RemoveContainer" containerID="c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"
Oct 30 11:52:14.551310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.551293    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:14.552149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.552086    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:52:14.553207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.553178    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:52:14.553314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.553258    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:52:14.553378 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.553307    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:52:14.806915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:14.806824    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:52:15.560498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:15.560444    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" event={"ID":"be30b482-dd34-4a19-a5b0-4813dc758914","Type":"ContainerStarted","Data":"6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3"}
Oct 30 11:52:15.763397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:15.763354    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:16.232817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:16.232780    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:52:16.233980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:16.233946    5060 patch_prober.go:29] interesting pod/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 11:52:16.234101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:16.234012    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 11:52:17.232362 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:17.232332    5060 patch_prober.go:29] interesting pod/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 11:52:17.232559 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:17.232380    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 11:52:17.728823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:17.728780    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:17.870071 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:17.870034    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:18.035153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:18.035055    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:18.231099 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:18.231058    5060 patch_prober.go:29] interesting pod/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 11:52:18.231315 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:18.231121    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 11:52:18.807476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:18.806431    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:18.967487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:18.965734    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:19.231011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:19.230942    5060 patch_prober.go:29] interesting pod/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 11:52:19.231193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:19.231016    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 11:52:19.857411 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:52:19.857369    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:52:19.927406 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:19.927310    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:20.231653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:20.231618    5060 patch_prober.go:29] interesting pod/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 11:52:20.231868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:20.231690    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 11:52:20.515110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:20.514314    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:52:21.236055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:21.236026    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:52:21.236596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:21.236574    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:52:21.237318 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:21.237298    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:52:21.853509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:21.853470    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:23.540431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:23.540388    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:23.698805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:23.698769    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:25.037892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:25.037849    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"]
Oct 30 11:52:25.048435 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:25.048400    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38c024-k7qgk"]
Oct 30 11:52:25.197389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:25.197335    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:25.387567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:25.376013    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:26.061021 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:26.060984    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:26.337647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:26.337569    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cb5328-fab9-41f4-90e6-cd185ae5feaa" path="/var/lib/kubelet/pods/36cb5328-fab9-41f4-90e6-cd185ae5feaa/volumes"
Oct 30 11:52:27.103144 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.103109    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:27.308526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.294586    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"]
Oct 30 11:52:27.308526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.296207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:27.311316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.309020    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d370553c-a314-4f75-93dd-ede1c9ac5372" containerName="revoke-keys"
Oct 30 11:52:27.311316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.309051    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d370553c-a314-4f75-93dd-ede1c9ac5372" containerName="revoke-keys"
Oct 30 11:52:27.311316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.309324    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d370553c-a314-4f75-93dd-ede1c9ac5372" containerName="revoke-keys"
Oct 30 11:52:27.316013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.314802    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.335778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.333729    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-84fed1\""
Oct 30 11:52:27.355352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.355264    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hsp\" (UniqueName: \"kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp\") pod \"payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.355769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.355742    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.386603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.384993    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-84fed1-dockercfg-cb4bn\""
Oct 30 11:52:27.401548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.401514    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"]
Oct 30 11:52:27.457012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.456980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hsp\" (UniqueName: \"kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp\") pod \"payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.457237 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.457097    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.458159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.458132    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.491498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.491427    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hsp\" (UniqueName: \"kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp\") pod \"payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:27.714792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:27.714759    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:28.028046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:28.028015    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"]
Oct 30 11:52:28.029219 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:52:28.029184    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca68d5ec_728b_4ced_9497_44a12e1144b2.slice/crio-8448274881a2bf49609084b0f15689dc78224ea52958a1c1e9af48c97c7ec5f8 WatchSource:0}: Error finding container 8448274881a2bf49609084b0f15689dc78224ea52958a1c1e9af48c97c7ec5f8: Status 404 returned error can't find the container with id 8448274881a2bf49609084b0f15689dc78224ea52958a1c1e9af48c97c7ec5f8
Oct 30 11:52:28.256828 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:28.256794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:28.738754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:28.738722    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689" event={"ID":"ca68d5ec-728b-4ced-9497-44a12e1144b2","Type":"ContainerStarted","Data":"64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217"}
Oct 30 11:52:28.738954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:28.738779    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689" event={"ID":"ca68d5ec-728b-4ced-9497-44a12e1144b2","Type":"ContainerStarted","Data":"8448274881a2bf49609084b0f15689dc78224ea52958a1c1e9af48c97c7ec5f8"}
Oct 30 11:52:28.738954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:28.738818    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:28.785393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:28.785329    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689" podStartSLOduration=1.7853111799999999 podStartE2EDuration="1.78531118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:28.784072565 +0000 UTC m=+1352.578957059" watchObservedRunningTime="2026-10-30 11:52:28.78531118 +0000 UTC m=+1352.580195676"
Oct 30 11:52:30.079494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:30.078733    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:31.832499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:31.832434    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:33.641106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:33.641054    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:34.046615 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:34.046579    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"]
Oct 30 11:52:34.056235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:34.056193    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca34736-q8x78"]
Oct 30 11:52:34.337104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:34.337012    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff4ae3f-4f4d-41ce-9356-ea273a700b84" path="/var/lib/kubelet/pods/3ff4ae3f-4f4d-41ce-9356-ea273a700b84/volumes"
Oct 30 11:52:35.477999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:35.477915    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:37.322738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:37.322693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:39.193055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:39.193017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:39.750492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:39.750446    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:52:40.968640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:40.968596    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:42.700442 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:42.700383    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:42.854154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:42.854113    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:44.445964 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:44.445931    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:45.183983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:45.183943    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:45.332914 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:45.332880    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:45.349897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:45.349853    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"]
Oct 30 11:52:45.363062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:45.363019    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58826b-698kh"]
Oct 30 11:52:45.477535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:45.475819    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:45.920494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:45.920378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:46.376800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:46.376753    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d491c8e3-da28-451c-8276-a0cbef2d0c42" path="/var/lib/kubelet/pods/d491c8e3-da28-451c-8276-a0cbef2d0c42/volumes"
Oct 30 11:52:48.070421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:48.070374    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:49.031566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:49.031528    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"]
Oct 30 11:52:49.044074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:49.044039    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad352c37-wczrr"]
Oct 30 11:52:49.313770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:49.313736    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:49.858762 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:52:49.858727    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:52:50.109356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:50.109259    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:50.337923 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:50.337887    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63acc2a-1b05-45b5-b364-1c9972732ddc" path="/var/lib/kubelet/pods/c63acc2a-1b05-45b5-b364-1c9972732ddc/volumes"
Oct 30 11:52:52.109239 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:52.109196    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:53.479395 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.479350    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7374-ae6196"]
Oct 30 11:52:53.485142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.485114    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:52:53.501175 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.501137    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7374-ae6196"]
Oct 30 11:52:53.506372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.506340    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\""
Oct 30 11:52:53.604144 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.604104    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vht5n\" (UniqueName: \"kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n\") pod \"test-curl-7374-ae6196\" (UID: \"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783\") " pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:52:53.705907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.705869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht5n\" (UniqueName: \"kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n\") pod \"test-curl-7374-ae6196\" (UID: \"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783\") " pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:52:53.733935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.733859    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht5n\" (UniqueName: \"kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n\") pod \"test-curl-7374-ae6196\" (UID: \"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783\") " pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:52:53.799703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.799664    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:52:53.838351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:53.838317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:54.106607 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:52:54.106575    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97f2eea8_2c37_42f9_8cb4_c3d7f78f8783.slice/crio-3487213ff85a1ba9ef30d5797b92dc03a57b32a92f6790f70e6a723db3361267 WatchSource:0}: Error finding container 3487213ff85a1ba9ef30d5797b92dc03a57b32a92f6790f70e6a723db3361267: Status 404 returned error can't find the container with id 3487213ff85a1ba9ef30d5797b92dc03a57b32a92f6790f70e6a723db3361267
Oct 30 11:52:54.109542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:54.109516    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7374-ae6196"]
Oct 30 11:52:54.241660 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:54.241628    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:55.043370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:55.043328    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7374-ae6196" event={"ID":"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783","Type":"ContainerStarted","Data":"352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64"}
Oct 30 11:52:55.043905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:55.043378    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7374-ae6196" event={"ID":"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783","Type":"ContainerStarted","Data":"3487213ff85a1ba9ef30d5797b92dc03a57b32a92f6790f70e6a723db3361267"}
Oct 30 11:52:55.085970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:55.085917    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7374-ae6196" podStartSLOduration=2.085898605 podStartE2EDuration="2.085898605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:55.082871933 +0000 UTC m=+1378.877756427" watchObservedRunningTime="2026-10-30 11:52:55.085898605 +0000 UTC m=+1378.880783105"
Oct 30 11:52:55.887343 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:55.887303    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:56.131488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:56.128327    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7374-ae6196"]
Oct 30 11:52:56.136062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:56.136026    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7374-ae6196"]
Oct 30 11:52:56.750570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:56.750531    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:56.889003 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:56.888971    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:56.905201 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:56.905163    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"]
Oct 30 11:52:56.916625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:56.916591    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93525b-mqlhj"]
Oct 30 11:52:57.038382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.038287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:57.095547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.095506    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7374-ae6196" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" containerName="test-curl-7374-ae6196" containerID="cri-o://352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64" gracePeriod=2
Oct 30 11:52:57.117158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.117119    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"]
Oct 30 11:52:57.117763 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.117748    5060 cpu_manager.go:414] "containerMap: removing container" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" containerName="test-curl-7374-ae6196"
Oct 30 11:52:57.117840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.117764    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" containerName="test-curl-7374-ae6196"
Oct 30 11:52:57.117896 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.117890    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" containerName="test-curl-7374-ae6196"
Oct 30 11:52:57.122660 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.122633    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.139496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.139416    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:52:57.139957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.139524    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:52:57.144891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.144863    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"]
Oct 30 11:52:57.161714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.161681    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:52:57.161903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.161722    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:52:57.190928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.190881    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28wr\" (UniqueName: \"kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.191260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.191240    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.191576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.191548    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.293380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.293279    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28wr\" (UniqueName: \"kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.293380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.293355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.293694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.293518    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.294126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.294101    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.296160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.296129    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.317734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.317699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28wr\" (UniqueName: \"kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr\") pod \"maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.442080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.442039    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:52:57.452117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.452076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:57.766283 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:52:57.753862    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b13414_e85a_4969_9681_4772d589c312.slice/crio-cf856ebd82f87c0764e2c3e7a0780c2e3d99e778a8e861cc7c4220f823149fe7 WatchSource:0}: Error finding container cf856ebd82f87c0764e2c3e7a0780c2e3d99e778a8e861cc7c4220f823149fe7: Status 404 returned error can't find the container with id cf856ebd82f87c0764e2c3e7a0780c2e3d99e778a8e861cc7c4220f823149fe7
Oct 30 11:52:57.766283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:57.756731    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"]
Oct 30 11:52:58.107806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:58.107764    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs" event={"ID":"15b13414-e85a-4969-9681-4772d589c312","Type":"ContainerStarted","Data":"cf856ebd82f87c0764e2c3e7a0780c2e3d99e778a8e861cc7c4220f823149fe7"}
Oct 30 11:52:58.112091 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:58.112055    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:58.339721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:58.339688    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d370553c-a314-4f75-93dd-ede1c9ac5372" path="/var/lib/kubelet/pods/d370553c-a314-4f75-93dd-ede1c9ac5372/volumes"
Oct 30 11:52:58.939039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:58.939001    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:52:59.122101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:59.122064    5060 generic.go:356] "Generic (PLEG): container finished" podID="15b13414-e85a-4969-9681-4772d589c312" containerID="cf3af293458df1dcd08af229a2883f6d448764bf37c1d32ffb467a73496417de" exitCode=0
Oct 30 11:52:59.122261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:59.122160    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs" event={"ID":"15b13414-e85a-4969-9681-4772d589c312","Type":"ContainerDied","Data":"cf3af293458df1dcd08af229a2883f6d448764bf37c1d32ffb467a73496417de"}
Oct 30 11:52:59.321362 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:59.321335    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:52:59.429132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:59.429097    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n\" (UniqueName: \"kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n\") pod \"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783\" (UID: \"97f2eea8-2c37-42f9-8cb4-c3d7f78f8783\") "
Oct 30 11:52:59.431505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:59.431463    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n" pod "97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" (UID: "97f2eea8-2c37-42f9-8cb4-c3d7f78f8783"). InnerVolumeSpecName "kube-api-access-vht5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:52:59.531118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:52:59.531028    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vht5n\" (UniqueName: \"kubernetes.io/projected/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783-kube-api-access-vht5n\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:00.132743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.132707    5060 generic.go:356] "Generic (PLEG): container finished" podID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" containerID="352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64" exitCode=137
Oct 30 11:53:00.132950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.132753    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:53:00.132950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.132809    5060 scope.go:122] "RemoveContainer" containerID="352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64"
Oct 30 11:53:00.145521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.145496    5060 scope.go:122] "RemoveContainer" containerID="352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64"
Oct 30 11:53:00.145896 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:00.145869    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64\": container with ID starting with 352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64 not found: ID does not exist" containerID="352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64"
Oct 30 11:53:00.145968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.145906    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64"} err="failed to get container status \"352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64\": rpc error: code = NotFound desc = could not find container \"352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64\": container with ID starting with 352c5d46d4e561feda54d475bb1b1ed8be02738c2d37199d694f2afd9e884d64 not found: ID does not exist"
Oct 30 11:53:00.156932 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:00.156896    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7374-ae6196\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:53:00.310929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.310898    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:53:00.337668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.337637    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" path="/var/lib/kubelet/pods/97f2eea8-2c37-42f9-8cb4-c3d7f78f8783/volumes"
Oct 30 11:53:00.342056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.342031    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca\") pod \"15b13414-e85a-4969-9681-4772d589c312\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") "
Oct 30 11:53:00.342187 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.342149    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr\" (UniqueName: \"kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr\") pod \"15b13414-e85a-4969-9681-4772d589c312\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") "
Oct 30 11:53:00.342187 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.342181    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca\") pod \"15b13414-e85a-4969-9681-4772d589c312\" (UID: \"15b13414-e85a-4969-9681-4772d589c312\") "
Oct 30 11:53:00.342635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.342611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca" pod "15b13414-e85a-4969-9681-4772d589c312" (UID: "15b13414-e85a-4969-9681-4772d589c312"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:00.342910 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.342727    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b13414-e85a-4969-9681-4772d589c312-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:00.344389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.344365    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca" pod "15b13414-e85a-4969-9681-4772d589c312" (UID: "15b13414-e85a-4969-9681-4772d589c312"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:53:00.344538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.344518    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr" pod "15b13414-e85a-4969-9681-4772d589c312" (UID: "15b13414-e85a-4969-9681-4772d589c312"). InnerVolumeSpecName "kube-api-access-q28wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:00.371651 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:00.371606    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7374-ae6196\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="97f2eea8-2c37-42f9-8cb4-c3d7f78f8783" pod="opendatahub/test-curl-7374-ae6196"
Oct 30 11:53:00.444860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.444768    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28wr\" (UniqueName: \"kubernetes.io/projected/15b13414-e85a-4969-9681-4772d589c312-kube-api-access-q28wr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:00.444860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.444816    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b13414-e85a-4969-9681-4772d589c312-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:00.730997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:00.730378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:01.031771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:01.031691    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"]
Oct 30 11:53:01.043959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:01.043927    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2291d5f1-9rjll"]
Oct 30 11:53:01.142673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:01.142640    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"
Oct 30 11:53:01.142673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:01.142653    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs" event={"ID":"15b13414-e85a-4969-9681-4772d589c312","Type":"ContainerDied","Data":"cf856ebd82f87c0764e2c3e7a0780c2e3d99e778a8e861cc7c4220f823149fe7"}
Oct 30 11:53:01.142909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:01.142707    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf856ebd82f87c0764e2c3e7a0780c2e3d99e778a8e861cc7c4220f823149fe7"
Oct 30 11:53:02.335661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:02.335625    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08653d8e-8504-43da-bf07-f5688268af6e" path="/var/lib/kubelet/pods/08653d8e-8504-43da-bf07-f5688268af6e/volumes"
Oct 30 11:53:02.508345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:02.508305    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:03.794808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:03.794774    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:03.939853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:03.939819    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:04.052480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.051869    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"]
Oct 30 11:53:04.053371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.053340    5060 cpu_manager.go:414] "containerMap: removing container" podUID="15b13414-e85a-4969-9681-4772d589c312" containerName="revoke-keys"
Oct 30 11:53:04.053625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.053607    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b13414-e85a-4969-9681-4772d589c312" containerName="revoke-keys"
Oct 30 11:53:04.053994 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.053976    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b13414-e85a-4969-9681-4772d589c312" containerName="revoke-keys"
Oct 30 11:53:04.060364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.060334    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.080748 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.080720    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:53:04.096653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.094990    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:53:04.096653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.095097    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:53:04.096653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.095183    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:53:04.139211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.135157    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"]
Oct 30 11:53:04.139211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.135311    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"]
Oct 30 11:53:04.139211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.135550    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" containerID="cri-o://f4219dd77f37bb7ca0e33681d78431542774fe981e7290536a733de220bdc629" gracePeriod=30
Oct 30 11:53:04.261286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.260478    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.261286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.260553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4w4\" (UniqueName: \"kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.261286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.260707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.305190 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.302379    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:04.366987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.366954    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.367487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.367466    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.367685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.367670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4w4\" (UniqueName: \"kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.369008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.368983    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.374268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.374237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.408330 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.408299    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4w4\" (UniqueName: \"kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4\") pod \"maas-api-revoke-sub-f932b170-b899m\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.567036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.566856    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:04.925322 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:53:04.925284    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f1d2a3_b390_4a74_9391_164f622a2378.slice/crio-b0df0c34a8a05c0a6045d11947d2a55150c2c76b49ada1538d3165fa5a3d4598 WatchSource:0}: Error finding container b0df0c34a8a05c0a6045d11947d2a55150c2c76b49ada1538d3165fa5a3d4598: Status 404 returned error can't find the container with id b0df0c34a8a05c0a6045d11947d2a55150c2c76b49ada1538d3165fa5a3d4598
Oct 30 11:53:04.926930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.926904    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"]
Oct 30 11:53:04.955350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.955316    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"]
Oct 30 11:53:04.960896 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.960876    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:04.975995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.975758    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:04.975995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.975831    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:04.976604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.976096    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kcrn\" (UniqueName: \"kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:04.976604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.976587    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:04.998482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:04.990585    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"]
Oct 30 11:53:05.077893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.077858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kcrn\" (UniqueName: \"kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.078098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.078005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.078098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.078054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.079079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.079048    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.081383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.081352    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.102390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.102355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kcrn\" (UniqueName: \"kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn\") pod \"maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.275624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.275589    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:05.319634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.319593    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m" event={"ID":"e8f1d2a3-b390-4a74-9391-164f622a2378","Type":"ContainerStarted","Data":"b0df0c34a8a05c0a6045d11947d2a55150c2c76b49ada1538d3165fa5a3d4598"}
Oct 30 11:53:05.657764 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:05.657614    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"]
Oct 30 11:53:06.377535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.376242    5060 generic.go:356] "Generic (PLEG): container finished" podID="e8f1d2a3-b390-4a74-9391-164f622a2378" containerID="b9c3a4d73e8fe8f2f194a6fa8e6dee722155ad8a39a5402b9b04c359f4229679" exitCode=0
Oct 30 11:53:06.399735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.386141    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m" event={"ID":"e8f1d2a3-b390-4a74-9391-164f622a2378","Type":"ContainerDied","Data":"b9c3a4d73e8fe8f2f194a6fa8e6dee722155ad8a39a5402b9b04c359f4229679"}
Oct 30 11:53:06.399735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.386211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm" event={"ID":"8c9c6877-28fe-4ea4-9623-371742e00b6c","Type":"ContainerStarted","Data":"4aac1330c8d331490f0a7fc92cb38e3d80213668a44ae1a133b2556f1f126598"}
Oct 30 11:53:06.399735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.386247    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm" event={"ID":"8c9c6877-28fe-4ea4-9623-371742e00b6c","Type":"ContainerStarted","Data":"8d0010e9f321388af25048db34cf7f29e091b246a62fbc71d6b89a8f07e20251"}
Oct 30 11:53:06.469681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.469046    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm" podStartSLOduration=2.469026045 podStartE2EDuration="2.469026045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:06.462339867 +0000 UTC m=+1390.257224372" watchObservedRunningTime="2026-10-30 11:53:06.469026045 +0000 UTC m=+1390.263910540"
Oct 30 11:53:06.828239 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.828115    5060 patch_prober.go:29] interesting pod/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:53:06.828427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:06.828279    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:53:07.441479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:07.438337    5060 generic.go:356] "Generic (PLEG): container finished" podID="8c9c6877-28fe-4ea4-9623-371742e00b6c" containerID="4aac1330c8d331490f0a7fc92cb38e3d80213668a44ae1a133b2556f1f126598" exitCode=0
Oct 30 11:53:07.441479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:07.439276    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm" event={"ID":"8c9c6877-28fe-4ea4-9623-371742e00b6c","Type":"ContainerDied","Data":"4aac1330c8d331490f0a7fc92cb38e3d80213668a44ae1a133b2556f1f126598"}
Oct 30 11:53:07.660478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:07.660188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:08.184531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.184357    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:08.265192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.264807    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:08.438880 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.438785    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:08.492675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.492640    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4\" (UniqueName: \"kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4\") pod \"e8f1d2a3-b390-4a74-9391-164f622a2378\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") "
Oct 30 11:53:08.493589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.493558    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca\") pod \"e8f1d2a3-b390-4a74-9391-164f622a2378\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") "
Oct 30 11:53:08.493816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.493798    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca\") pod \"e8f1d2a3-b390-4a74-9391-164f622a2378\" (UID: \"e8f1d2a3-b390-4a74-9391-164f622a2378\") "
Oct 30 11:53:08.495648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.495623    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca" pod "e8f1d2a3-b390-4a74-9391-164f622a2378" (UID: "e8f1d2a3-b390-4a74-9391-164f622a2378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:08.518126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.518093    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"
Oct 30 11:53:08.519114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.519080    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4" pod "e8f1d2a3-b390-4a74-9391-164f622a2378" (UID: "e8f1d2a3-b390-4a74-9391-164f622a2378"). InnerVolumeSpecName "kube-api-access-md4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:08.524140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.524109    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca" pod "e8f1d2a3-b390-4a74-9391-164f622a2378" (UID: "e8f1d2a3-b390-4a74-9391-164f622a2378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:53:08.599585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.598046    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m" event={"ID":"e8f1d2a3-b390-4a74-9391-164f622a2378","Type":"ContainerDied","Data":"b0df0c34a8a05c0a6045d11947d2a55150c2c76b49ada1538d3165fa5a3d4598"}
Oct 30 11:53:08.599585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.598110    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0df0c34a8a05c0a6045d11947d2a55150c2c76b49ada1538d3165fa5a3d4598"
Oct 30 11:53:08.614874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.614778    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:08.614874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.614816    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4w4\" (UniqueName: \"kubernetes.io/projected/e8f1d2a3-b390-4a74-9391-164f622a2378-kube-api-access-md4w4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:08.614874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:08.614833    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f1d2a3-b390-4a74-9391-164f622a2378-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:09.592678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.592548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm" event={"ID":"8c9c6877-28fe-4ea4-9623-371742e00b6c","Type":"ContainerDied","Data":"8d0010e9f321388af25048db34cf7f29e091b246a62fbc71d6b89a8f07e20251"}
Oct 30 11:53:09.593265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.593228    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0010e9f321388af25048db34cf7f29e091b246a62fbc71d6b89a8f07e20251"
Oct 30 11:53:09.608352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.608310    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:09.663483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.662714    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:09.937680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.937569    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:09.943483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.940481    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn\" (UniqueName: \"kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn\") pod \"8c9c6877-28fe-4ea4-9623-371742e00b6c\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") "
Oct 30 11:53:09.943802 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.943773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca\") pod \"8c9c6877-28fe-4ea4-9623-371742e00b6c\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") "
Oct 30 11:53:09.943987 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.943966    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca\") pod \"8c9c6877-28fe-4ea4-9623-371742e00b6c\" (UID: \"8c9c6877-28fe-4ea4-9623-371742e00b6c\") "
Oct 30 11:53:09.972926 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.953922    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:09.981491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:09.973863    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca" pod "8c9c6877-28fe-4ea4-9623-371742e00b6c" (UID: "8c9c6877-28fe-4ea4-9623-371742e00b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:10.015979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.010806    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn" pod "8c9c6877-28fe-4ea4-9623-371742e00b6c" (UID: "8c9c6877-28fe-4ea4-9623-371742e00b6c"). InnerVolumeSpecName "kube-api-access-7kcrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:10.040493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.028580    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca" pod "8c9c6877-28fe-4ea4-9623-371742e00b6c" (UID: "8c9c6877-28fe-4ea4-9623-371742e00b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:53:10.076493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.066064    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:10.076493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.069137    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:10.076493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.069164    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kcrn\" (UniqueName: \"kubernetes.io/projected/8c9c6877-28fe-4ea4-9623-371742e00b6c-kube-api-access-7kcrn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:10.076493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.069180    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9c6877-28fe-4ea4-9623-371742e00b6c-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:10.097618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.097578    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:10.677845 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.637654    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:10.779624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.769793    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:10.869134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.869097    5060 generic.go:356] "Generic (PLEG): container finished" podID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerID="f4219dd77f37bb7ca0e33681d78431542774fe981e7290536a733de220bdc629" exitCode=0
Oct 30 11:53:10.869518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.869497    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"
Oct 30 11:53:10.873118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.873091    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" event={"ID":"dbac8954-3288-4d1d-95a6-a29b04e45490","Type":"ContainerDied","Data":"f4219dd77f37bb7ca0e33681d78431542774fe981e7290536a733de220bdc629"}
Oct 30 11:53:10.873307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.873292    5060 scope.go:122] "RemoveContainer" containerID="60637fbc17a8c99ec1681760cec5529a22eb8b3b01d657abc74689c19948aca3"
Oct 30 11:53:10.966503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.954561    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:53:10.966503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.955302    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:53:10.966503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.958178    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:10.968941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:10.966970    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:53:11.006663 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.006554    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:11.058475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.056609    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:11.058475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.057047    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:53:11.113487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.113027    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:11.158836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.117491    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:53:11.158836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.119713    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:11.158836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.147527    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:11.403040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.402999    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 11:53:11.596533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.590793    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:53:11.710479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.645477    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.738627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.738576    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.739346 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.739325    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.739845 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.739826    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.754417 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.754389    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.755130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.755109    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.778221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.778193    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.778388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.778371    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.778544 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.778527    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo\") pod \"dbac8954-3288-4d1d-95a6-a29b04e45490\" (UID: \"dbac8954-3288-4d1d-95a6-a29b04e45490\") "
Oct 30 11:53:11.806699 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.713107    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.809173    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.647042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.739235    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.739671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.750137    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "kube-api-access-6wb4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.776652    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.780997    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.781228    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.771546    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 11:53:11.824479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.824034    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:53:11.853075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.845323    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:11.882936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.882169    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo" pod "dbac8954-3288-4d1d-95a6-a29b04e45490" (UID: "dbac8954-3288-4d1d-95a6-a29b04e45490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:53:11.891252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.891218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979539    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wb4d\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-kube-api-access-6wb4d\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979573    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979590    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979603    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979617    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979631    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979644    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbac8954-3288-4d1d-95a6-a29b04e45490-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979658    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.979670    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbac8954-3288-4d1d-95a6-a29b04e45490-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:11.993575    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:12.046028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.024013    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:53:12.078111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.061162    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 11:53:12.130021 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.129882    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 11:53:12.131335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.131298    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:53:12.159912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.132301    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:53:12.161203 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.158800    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:12.161892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.161870    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:53:12.264317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.264280    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9" event={"ID":"dbac8954-3288-4d1d-95a6-a29b04e45490","Type":"ContainerDied","Data":"aad9e10eee549a73ceda703329e91dac53a8eae116e45f50dfb4e7f213643a38"}
Oct 30 11:53:12.265139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.265104    5060 scope.go:122] "RemoveContainer" containerID="f4219dd77f37bb7ca0e33681d78431542774fe981e7290536a733de220bdc629"
Oct 30 11:53:12.266526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.266507    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 11:53:12.285689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.267545    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"
Oct 30 11:53:12.286876 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.286025    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:53:12.418685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.418218    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:12.441994 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.441963    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:53:12.445512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.445443    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:53:12.450289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.450249    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:53:12.551016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.550034    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 11:53:12.636687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.633713    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:53:12.695564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.695267    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:53:12.695564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.695329    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:53:12.695564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.695362    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 11:53:12.695564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.695393    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:53:12.756965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.756938    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"]
Oct 30 11:53:12.757652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.757631    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy"
Oct 30 11:53:12.757794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.757783    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy"
Oct 30 11:53:12.757857 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.757850    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8c9c6877-28fe-4ea4-9623-371742e00b6c" containerName="revoke-keys"
Oct 30 11:53:12.757908 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.757901    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9c6877-28fe-4ea4-9623-371742e00b6c" containerName="revoke-keys"
Oct 30 11:53:12.757965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.757958    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e8f1d2a3-b390-4a74-9391-164f622a2378" containerName="revoke-keys"
Oct 30 11:53:12.758013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.758005    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f1d2a3-b390-4a74-9391-164f622a2378" containerName="revoke-keys"
Oct 30 11:53:12.758204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.758196    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f1d2a3-b390-4a74-9391-164f622a2378" containerName="revoke-keys"
Oct 30 11:53:12.758268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.758261    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy"
Oct 30 11:53:12.758317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.758311    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy"
Oct 30 11:53:12.758360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.758354    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9c6877-28fe-4ea4-9623-371742e00b6c" containerName="revoke-keys"
Oct 30 11:53:12.763447 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.763422    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.768446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.768424    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"]
Oct 30 11:53:12.781851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.781821    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6d9258eb-openshift-default-dockercfg-tx4ln\""
Oct 30 11:53:12.866689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.861299    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.867165 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.867138    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.867406 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.867389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.867695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.867676    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.867867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.867853    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.868005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.867991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.868208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.868194    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.868384 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.868370    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.868581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.868558    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t497\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.969952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.969868    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.969952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.969915    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970447 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970425    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970558    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970593    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970658    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970316    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970707    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970753    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t497\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.970918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.970341    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.971184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.971159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.971388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.971369    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.971931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.971912    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.973673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.973657    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:12.973986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:12.973963    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:13.039183 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.030536    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:13.039183 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.039100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t497\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:13.111754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.111714    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:13.331287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.331224    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"]
Oct 30 11:53:13.337235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.336890    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-93525b-openshift-default-7c9b64476f-q49b9"]
Oct 30 11:53:13.518849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.518811    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:13.518959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.518897    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:13.519008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.518961    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:13.519635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.519610    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"]
Oct 30 11:53:13.861001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:13.860964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:14.348808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:14.348769    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" path="/var/lib/kubelet/pods/dbac8954-3288-4d1d-95a6-a29b04e45490/volumes"
Oct 30 11:53:14.353373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:14.353330    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" event={"ID":"521369ae-35c1-4453-a4ac-f8b19403726a","Type":"ContainerStarted","Data":"c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a"}
Oct 30 11:53:14.353602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:14.353405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" event={"ID":"521369ae-35c1-4453-a4ac-f8b19403726a","Type":"ContainerStarted","Data":"62b974731cc3baf7ef286b627c1d9f6ffc29d68b99b74ceca917bb224287849e"}
Oct 30 11:53:14.401030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:14.400972    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podStartSLOduration=2.4009550649999998 podStartE2EDuration="2.400955065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:14.398468763 +0000 UTC m=+1398.193353254" watchObservedRunningTime="2026-10-30 11:53:14.400955065 +0000 UTC m=+1398.195839558"
Oct 30 11:53:15.112708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.112670    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:15.114255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.114221    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 11:53:15.114427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.114289    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 11:53:15.506478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.505499    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:15.608485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.598562    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:53:15.608485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.598625    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:53:15.647358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.647313    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:15.810761 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:15.810218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:16.117579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:16.115277    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 11:53:16.117579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:16.115336    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 11:53:17.113480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:17.112996    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 11:53:17.113480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:17.113057    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 11:53:17.437551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:17.436700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:18.136515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.127646    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 11:53:18.136515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.127704    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 11:53:18.210645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.210606    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:18.439744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.439153    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:18.485489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.477145    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"]
Oct 30 11:53:18.540570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.540526    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d9258eb-rkhhs"]
Oct 30 11:53:18.636849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:18.636809    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:19.034996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:19.034956    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:19.113860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:19.113824    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 11:53:19.114120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:19.114094    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 11:53:19.860052 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:19.860018    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:53:20.113209 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.113093    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 11:53:20.113209 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.113163    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 11:53:20.339428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.339387    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b13414-e85a-4969-9681-4772d589c312" path="/var/lib/kubelet/pods/15b13414-e85a-4969-9681-4772d589c312/volumes"
Oct 30 11:53:20.531289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.531255    5060 generic.go:356] "Generic (PLEG): container finished" podID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerID="9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155" exitCode=0
Oct 30 11:53:20.531545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.531323    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" event={"ID":"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76","Type":"ContainerDied","Data":"9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"}
Oct 30 11:53:20.531809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.531793    5060 scope.go:122] "RemoveContainer" containerID="9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"
Oct 30 11:53:20.533389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.533361    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:20.533518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.533426    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:20.533518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.533489    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:20.661474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.661403    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:20.860161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:20.860062    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:21.270643 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.270614    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:21.299059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.298907    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:21.322226 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.322192    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:53:21.548543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.545591    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" event={"ID":"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76","Type":"ContainerStarted","Data":"094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a"}
Oct 30 11:53:21.614737 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.614698    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:21.617338 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.617310    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:53:21.617765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.617741    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 11:53:21.617954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:21.617930    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 11:53:22.127590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:22.114671    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:22.381540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:22.376750    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:22.618109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:22.618070    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 11:53:22.618411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:22.618378    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 11:53:22.792480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:22.790899    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:23.624479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:23.624359    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 11:53:23.624479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:23.624435    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 11:53:24.107429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:24.107378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:24.617588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:24.617554    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 11:53:24.617781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:24.617615    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 11:53:24.670664 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:24.670614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:25.617583 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:25.617539    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 11:53:25.617808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:25.617610    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 11:53:25.969704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:25.969667    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:26.206167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:26.206039    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:26.617891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:26.617850    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 11:53:26.618116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:26.617948    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 11:53:26.752439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:26.752376    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:27.318722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.318685    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"]
Oct 30 11:53:27.417421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.417387    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:27.487142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.487101    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"]
Oct 30 11:53:27.488334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.488295    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy"
Oct 30 11:53:27.488334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.488328    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbac8954-3288-4d1d-95a6-a29b04e45490" containerName="istio-proxy"
Oct 30 11:53:27.492789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.492766    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.512322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.512292    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"]
Oct 30 11:53:27.532837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.532802    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"]
Oct 30 11:53:27.540879 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.540849    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.569680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.569617    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:27.574976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.574946    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"]
Oct 30 11:53:27.617509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617430    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.617723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617588    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.617723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617647    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlth\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.617817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617736    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.617817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617773    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.617885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617870    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.617975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617896    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.618016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617974    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.618016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.617995    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf854\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.618119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618017    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.618119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618084    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.618228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.618228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.618334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.618411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618397    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.618499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618484    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.618610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.618661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.618647    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.622330 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.622308    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:53:27.622847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.622826    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:53:27.623689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.623671    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:53:27.723902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.723749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.767989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768050    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768096    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768324    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768360    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768490    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768534    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768635    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlth\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768756    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768824    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768848    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768918    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf854\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.768977    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.769834    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.770048    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.772527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.770373    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.776552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.773968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.776552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.774187    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.776552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.774404    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.776552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.774658    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.776552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.774871    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.776552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.775109    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.777293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.777249    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.780769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.780719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.782194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.782172    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.828334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.810264    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.828334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.826272    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.828334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.826948    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf854\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:27.833557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.829089    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlth\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.833557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.831327    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.843477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.840546    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:27.869017 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:27.868973    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:28.123428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.123339    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:28.406586 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:53:28.406539    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22894126_e4c1_4cb1_acd1_2819b70eafe5.slice/crio-57bd338f373a823781363b4443836e425b5756ff2f8520496cd7729a209743fb WatchSource:0}: Error finding container 57bd338f373a823781363b4443836e425b5756ff2f8520496cd7729a209743fb: Status 404 returned error can't find the container with id 57bd338f373a823781363b4443836e425b5756ff2f8520496cd7729a209743fb
Oct 30 11:53:28.409290 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.409263    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"]
Oct 30 11:53:28.409695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.409661    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:28.409785 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.409734    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:28.409785 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.409772    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:28.481667 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:53:28.481640    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4c8b8c_db57_46ce_911c_50372beca916.slice/crio-7bbafe13736a5a808d8c99b15216c29181ed51218e99202738da0c95ed2fa010 WatchSource:0}: Error finding container 7bbafe13736a5a808d8c99b15216c29181ed51218e99202738da0c95ed2fa010: Status 404 returned error can't find the container with id 7bbafe13736a5a808d8c99b15216c29181ed51218e99202738da0c95ed2fa010
Oct 30 11:53:28.484502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.484468    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:28.484622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.484526    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"]
Oct 30 11:53:28.484622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.484548    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:28.484622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.484603    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:53:28.660574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.659934    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" event={"ID":"ef4c8b8c-db57-46ce-911c-50372beca916","Type":"ContainerStarted","Data":"4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06"}
Oct 30 11:53:28.660574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.660019    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" event={"ID":"ef4c8b8c-db57-46ce-911c-50372beca916","Type":"ContainerStarted","Data":"7bbafe13736a5a808d8c99b15216c29181ed51218e99202738da0c95ed2fa010"}
Oct 30 11:53:28.671059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.668902    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:28.681475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.675513    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" event={"ID":"22894126-e4c1-4cb1-acd1-2819b70eafe5","Type":"ContainerStarted","Data":"0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f"}
Oct 30 11:53:28.681475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.675622    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" event={"ID":"22894126-e4c1-4cb1-acd1-2819b70eafe5","Type":"ContainerStarted","Data":"57bd338f373a823781363b4443836e425b5756ff2f8520496cd7729a209743fb"}
Oct 30 11:53:28.726379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.726314    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podStartSLOduration=1.726295345 podStartE2EDuration="1.726295345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:28.708681316 +0000 UTC m=+1412.503565810" watchObservedRunningTime="2026-10-30 11:53:28.726295345 +0000 UTC m=+1412.521179845"
Oct 30 11:53:28.869673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:28.869633    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:29.130479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:29.129891    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:29.134968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:29.134572    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:29.134968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:29.134648    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:29.871064 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:29.871033    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 11:53:29.871539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:29.871503    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 11:53:30.106103 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.106069    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:30.132015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.131918    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:30.132301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.132271    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:30.257067 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.257039    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:30.856000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.855926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:30.869967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.869937    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 11:53:30.870141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:30.869987    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 11:53:31.136505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:31.135647    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:31.136505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:31.135705    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:31.850486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:31.849904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:31.869486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:31.869429    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 11:53:31.869703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:31.869514    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 11:53:32.029481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.026359    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:32.218687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.218660    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:32.231438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.223639    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:32.328211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.328172    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z" podUID="e98568c1-21ca-4523-bb43-0fa8e383d5b7" containerName="payload-pre-processing" containerID="cri-o://0e01b8d9fcb1b7da7073aa190cf20dc972ba05996a4ca08cbb9d6d56a359d487" gracePeriod=25
Oct 30 11:53:32.797989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.776316    5060 generic.go:356] "Generic (PLEG): container finished" podID="e98568c1-21ca-4523-bb43-0fa8e383d5b7" containerID="0e01b8d9fcb1b7da7073aa190cf20dc972ba05996a4ca08cbb9d6d56a359d487" exitCode=0
Oct 30 11:53:32.797989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.776652    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z" event={"ID":"e98568c1-21ca-4523-bb43-0fa8e383d5b7","Type":"ContainerDied","Data":"0e01b8d9fcb1b7da7073aa190cf20dc972ba05996a4ca08cbb9d6d56a359d487"}
Oct 30 11:53:32.871525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.871487    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 11:53:32.871705 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.871546    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 11:53:32.937109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.912582    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:32.978485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:32.977600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:53:33.048477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.047569    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podStartSLOduration=6.047551617 podStartE2EDuration="6.047551617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:28.771836853 +0000 UTC m=+1412.566721347" watchObservedRunningTime="2026-10-30 11:53:33.047551617 +0000 UTC m=+1416.842436113"
Oct 30 11:53:33.086481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.086001    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw\" (UniqueName: \"kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw\") pod \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") "
Oct 30 11:53:33.086481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.086146    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume\") pod \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\" (UID: \"e98568c1-21ca-4523-bb43-0fa8e383d5b7\") "
Oct 30 11:53:33.110540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.100374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume" pod "e98568c1-21ca-4523-bb43-0fa8e383d5b7" (UID: "e98568c1-21ca-4523-bb43-0fa8e383d5b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:33.147040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.137976    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:33.147040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.138055    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:33.150608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.149941    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw" pod "e98568c1-21ca-4523-bb43-0fa8e383d5b7" (UID: "e98568c1-21ca-4523-bb43-0fa8e383d5b7"). InnerVolumeSpecName "kube-api-access-vrrdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:33.198479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.188866    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98568c1-21ca-4523-bb43-0fa8e383d5b7-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:33.198479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.188913    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrrdw\" (UniqueName: \"kubernetes.io/projected/e98568c1-21ca-4523-bb43-0fa8e383d5b7-kube-api-access-vrrdw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:33.878480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.863147    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z" event={"ID":"e98568c1-21ca-4523-bb43-0fa8e383d5b7","Type":"ContainerDied","Data":"71426d1e04e1945c31caa39dd3ea81bb63018d008078ff902a8fd38f68cf9bb5"}
Oct 30 11:53:33.878480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.863349    5060 scope.go:122] "RemoveContainer" containerID="0e01b8d9fcb1b7da7073aa190cf20dc972ba05996a4ca08cbb9d6d56a359d487"
Oct 30 11:53:33.878480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.863712    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"
Oct 30 11:53:33.906480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.902798    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 11:53:33.906480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:33.902877    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 11:53:34.139535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.136573    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:34.139535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.136654    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:34.139797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.139591    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"]
Oct 30 11:53:34.156507 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.156439    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-868569b7f-t7b7z"]
Oct 30 11:53:34.356611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.356578    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98568c1-21ca-4523-bb43-0fa8e383d5b7" path="/var/lib/kubelet/pods/e98568c1-21ca-4523-bb43-0fa8e383d5b7/volumes"
Oct 30 11:53:34.867478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.867428    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:34.870310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.870286    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 11:53:34.870479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:34.870335    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 11:53:35.128738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:35.128637    5060 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 11:53:35.129247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:35.128732    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 11:53:35.874326 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:35.874293    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:35.874700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:35.874672    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:35.875398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:35.875380    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:53:36.051978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:36.051941    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:36.133049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:36.132973    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:36.133466 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:36.133428    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:36.134107 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:36.134091    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:53:36.233960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:36.233926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:37.043130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:37.043083    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:37.779918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:37.779879    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:37.936343 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:37.936313    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:38.700259 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:38.700218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:38.865481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:38.863768    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:40.126213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:40.126175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:42.169281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:42.169241    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:43.034639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:43.034592    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"]
Oct 30 11:53:43.046153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:43.046118    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848ac2b9-v7sbn"]
Oct 30 11:53:43.801222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:43.801183    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:44.404332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:44.404295    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e22cf4-237e-4b04-8818-aca99a25e554" path="/var/lib/kubelet/pods/58e22cf4-237e-4b04-8818-aca99a25e554/volumes"
Oct 30 11:53:44.593893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:44.593842    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:44.692012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:44.691910    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"]
Oct 30 11:53:44.704552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:44.704510    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7bf55-7kftm"]
Oct 30 11:53:44.738065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:44.738030    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:44.879955 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:44.879921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:45.259012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:45.258977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:46.336198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:46.336164    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9c6877-28fe-4ea4-9623-371742e00b6c" path="/var/lib/kubelet/pods/8c9c6877-28fe-4ea4-9623-371742e00b6c/volumes"
Oct 30 11:53:46.644122 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:46.644016    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:48.412239 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:48.412205    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:48.578849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:48.578808    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:48.745638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:48.745605    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:49.428132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:49.428090    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:49.861255 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:49.861217    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:53:50.413469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:50.413420    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:51.831719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:51.831663    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:52.033926 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:52.033889    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:52.181533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:52.177877    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"]
Oct 30 11:53:52.181533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:52.178156    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy" containerID="cri-o://6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3" gracePeriod=30
Oct 30 11:53:52.209670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:52.209629    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:52.487883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:52.487849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:53.554638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:53.554596    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:54.076260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:54.076220    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:54.325521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:54.320632    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:54.473771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:54.473731    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:54.624818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:54.624779    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:56.337122 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:56.337088    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:56.762873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:56.762843    5060 scope.go:122] "RemoveContainer" containerID="4ba2da578c2c5533e269ad11ee48f2c0fb253d9e9e1a21b86920dfb0bf45e79b"
Oct 30 11:53:56.774055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:56.774031    5060 scope.go:122] "RemoveContainer" containerID="970b9f1fae18716c6967028d02a4bf82bbfa0f992f35d0924139fe370319fe05"
Oct 30 11:53:56.786256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:56.786217    5060 scope.go:122] "RemoveContainer" containerID="785f98f83947cc03176f6f0a40eff5e59e139e30c9f44a5940d57737c3161345"
Oct 30 11:53:57.491068 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.491027    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:53:57.529526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529490    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.529732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529609    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.529732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529678    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.529732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529729    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.529907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529790    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.529907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.529907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529865    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.530065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529935    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.530065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529965    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:57.530065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:57.530065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.529989    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token\") pod \"be30b482-dd34-4a19-a5b0-4813dc758914\" (UID: \"be30b482-dd34-4a19-a5b0-4813dc758914\") "
Oct 30 11:53:57.530266 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530246    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:57.530389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:57.530497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530371    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:57.530949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530885    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be30b482-dd34-4a19-a5b0-4813dc758914-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.530949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530906    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.530949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530920    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.530949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530934    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.530949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.530949    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.532243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.532218    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "kube-api-access-n6mlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:57.532412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.532388    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:53:57.532709 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.532683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:57.532823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.532768    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token" pod "be30b482-dd34-4a19-a5b0-4813dc758914" (UID: "be30b482-dd34-4a19-a5b0-4813dc758914"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:57.632707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.632670    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-kube-api-access-n6mlq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.632707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.632705    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be30b482-dd34-4a19-a5b0-4813dc758914-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.632707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.632715    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be30b482-dd34-4a19-a5b0-4813dc758914-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:57.632965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:57.632724    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be30b482-dd34-4a19-a5b0-4813dc758914-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:53:58.220817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.220758    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:58.271744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.271703    5060 generic.go:356] "Generic (PLEG): container finished" podID="be30b482-dd34-4a19-a5b0-4813dc758914" containerID="6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3" exitCode=0
Oct 30 11:53:58.271939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.271849    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" event={"ID":"be30b482-dd34-4a19-a5b0-4813dc758914","Type":"ContainerDied","Data":"6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3"}
Oct 30 11:53:58.271939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.271919    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr" event={"ID":"be30b482-dd34-4a19-a5b0-4813dc758914","Type":"ContainerDied","Data":"e8bd6e24d7810b7a36fc023260d6daf221e8fb0c4196ea222dfab5199d3e4001"}
Oct 30 11:53:58.272095 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.271942    5060 scope.go:122] "RemoveContainer" containerID="6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3"
Oct 30 11:53:58.272149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.272099    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"
Oct 30 11:53:58.296585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.296546    5060 scope.go:122] "RemoveContainer" containerID="c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"
Oct 30 11:53:58.331587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.330980    5060 scope.go:122] "RemoveContainer" containerID="6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3"
Oct 30 11:53:58.331587 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:58.331521    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3\": container with ID starting with 6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3 not found: ID does not exist" containerID="6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3"
Oct 30 11:53:58.331587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.331558    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3"} err="failed to get container status \"6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3\": rpc error: code = NotFound desc = could not find container \"6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3\": container with ID starting with 6646ee1444ebcbb2f15a7686e140475934cfa70285ef2e21045c7cdb661ad0a3 not found: ID does not exist"
Oct 30 11:53:58.331587 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.331581    5060 scope.go:122] "RemoveContainer" containerID="c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"
Oct 30 11:53:58.332061 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:53:58.332038    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f\": container with ID starting with c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f not found: ID does not exist" containerID="c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"
Oct 30 11:53:58.332158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.332069    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f"} err="failed to get container status \"c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f\": rpc error: code = NotFound desc = could not find container \"c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f\": container with ID starting with c84b292283c6dbcf5755ae6b55e96f3b2a9796048e0cd50f92d7f3be02ee085f not found: ID does not exist"
Oct 30 11:53:58.348346 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.348311    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"]
Oct 30 11:53:58.352529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.352498    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b7bf55-openshift-default-69cf9bbfb5-8qsnr"]
Oct 30 11:53:58.492960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.492865    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:58.649503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:58.649470    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:53:59.053350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:53:59.053306    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:00.318737 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.318705    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:00.336334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.336301    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" path="/var/lib/kubelet/pods/be30b482-dd34-4a19-a5b0-4813dc758914/volumes"
Oct 30 11:54:00.455866 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.455833    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"]
Oct 30 11:54:00.456433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456420    5060 cpu_manager.go:414] "containerMap: removing container" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy"
Oct 30 11:54:00.456509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456434    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy"
Oct 30 11:54:00.456509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456445    5060 cpu_manager.go:414] "containerMap: removing container" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy"
Oct 30 11:54:00.456509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456467    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy"
Oct 30 11:54:00.456509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456490    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e98568c1-21ca-4523-bb43-0fa8e383d5b7" containerName="payload-pre-processing"
Oct 30 11:54:00.456509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456497    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98568c1-21ca-4523-bb43-0fa8e383d5b7" containerName="payload-pre-processing"
Oct 30 11:54:00.456687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456619    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98568c1-21ca-4523-bb43-0fa8e383d5b7" containerName="payload-pre-processing"
Oct 30 11:54:00.456687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456632    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy"
Oct 30 11:54:00.456687 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.456639    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="be30b482-dd34-4a19-a5b0-4813dc758914" containerName="istio-proxy"
Oct 30 11:54:00.461835 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.461811    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.485935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.485221    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c92679-dockercfg-7s569\""
Oct 30 11:54:00.485935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.485891    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c92679\""
Oct 30 11:54:00.490393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.490341    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:00.511298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.511263    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"]
Oct 30 11:54:00.597536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.592404    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2hz\" (UniqueName: \"kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz\") pod \"payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.597536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.592664    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.654396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.654362    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"]
Oct 30 11:54:00.679882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.679843    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:00.699561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.699523    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.700051 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.700027    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2hz\" (UniqueName: \"kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz\") pod \"payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.822515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.702618    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.822515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.801221    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"]
Oct 30 11:54:00.822515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.809519    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2hz\" (UniqueName: \"kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz\") pod \"payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:00.914158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.914048    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tjh\" (UniqueName: \"kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh\") pod \"payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:00.914158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:00.914127    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:01.015752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.015712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tjh\" (UniqueName: \"kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh\") pod \"payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:01.015942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.015784    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:01.016601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.016573    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:01.043488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.043011    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tjh\" (UniqueName: \"kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh\") pod \"payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:01.082477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.081734    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:01.109647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.108064    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:01.337284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.337250    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:01.502868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.502835    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"]
Oct 30 11:54:01.511743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.511694    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"]
Oct 30 11:54:01.806375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:01.806260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:02.319537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.319477    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296" event={"ID":"5f9f1ab8-0bb2-42a8-9013-f871ac89900a","Type":"ContainerStarted","Data":"22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc"}
Oct 30 11:54:02.319788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.319571    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296" event={"ID":"5f9f1ab8-0bb2-42a8-9013-f871ac89900a","Type":"ContainerStarted","Data":"6b85d48dabfc908ac2b9aa5f8b8a5486b4aa9c4c693c0d3a4cffe3f5d092430f"}
Oct 30 11:54:02.319788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.319615    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:02.322404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.322367    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b" event={"ID":"fe0b62cc-764b-41b5-96c7-70d9832e6fe0","Type":"ContainerStarted","Data":"ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a"}
Oct 30 11:54:02.322604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.322425    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b" event={"ID":"fe0b62cc-764b-41b5-96c7-70d9832e6fe0","Type":"ContainerStarted","Data":"df4db07cc9e1a0cef9503832a6af502e9a0f79dea4653ea8ff2aeda7106b5717"}
Oct 30 11:54:02.322604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.322497    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:02.362140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.362085    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296" podStartSLOduration=2.362069327 podStartE2EDuration="2.362069327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:02.360102435 +0000 UTC m=+1446.154986930" watchObservedRunningTime="2026-10-30 11:54:02.362069327 +0000 UTC m=+1446.156953902"
Oct 30 11:54:02.405195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.405118    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b" podStartSLOduration=2.405095403 podStartE2EDuration="2.405095403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:02.400821626 +0000 UTC m=+1446.195706120" watchObservedRunningTime="2026-10-30 11:54:02.405095403 +0000 UTC m=+1446.199979898"
Oct 30 11:54:02.594051 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:02.593647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:03.242197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:03.242155    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:04.286639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.286604    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"]
Oct 30 11:54:04.287134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.286813    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy" containerID="cri-o://c0e6f38c8d0815ae0faec50a9a5883d3986f1a7b9cca45910b2e3d19eada0285" gracePeriod=30
Oct 30 11:54:04.301273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.301239    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"]
Oct 30 11:54:04.301541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.301517    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" containerID="cri-o://f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3" gracePeriod=30
Oct 30 11:54:04.307984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.307954    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"]
Oct 30 11:54:04.308259 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.308235    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy" containerID="cri-o://806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68" gracePeriod=30
Oct 30 11:54:04.313521 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.313486    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"]
Oct 30 11:54:04.313855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.313826    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy" containerID="cri-o://2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9" gracePeriod=30
Oct 30 11:54:04.477618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.477575    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:54:04.477800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:04.477642    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:54:05.261493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:05.261413    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:07.094284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:07.094246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:08.682976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:08.682937    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:08.830297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:08.830255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:08.999338 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:08.999298    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"]
Oct 30 11:54:08.999711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:08.999682    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy" containerID="cri-o://0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f" gracePeriod=30
Oct 30 11:54:09.033570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.018304    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"]
Oct 30 11:54:09.033570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.019049    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy" containerID="cri-o://c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a" gracePeriod=30
Oct 30 11:54:09.033570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.027763    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"]
Oct 30 11:54:09.033570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.028059    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy" containerID="cri-o://4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06" gracePeriod=30
Oct 30 11:54:09.046272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.046240    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:09.335540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.332075    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:09.441016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.440982    5060 generic.go:356] "Generic (PLEG): container finished" podID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerID="c0e6f38c8d0815ae0faec50a9a5883d3986f1a7b9cca45910b2e3d19eada0285" exitCode=0
Oct 30 11:54:09.441162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.441079    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" event={"ID":"af2c3a63-dd7a-43cc-96a2-7186fd13763e","Type":"ContainerDied","Data":"c0e6f38c8d0815ae0faec50a9a5883d3986f1a7b9cca45910b2e3d19eada0285"}
Oct 30 11:54:09.625647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.620319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:09.835469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.832811    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:09.907375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.906943    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.997620    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.997719    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.997762    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.997813    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.997887    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.997926    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.998085    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.998125    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.004515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:09.998171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo\") pod \"cdc77535-bdd2-46bf-b536-d32ae91d3043\" (UID: \"cdc77535-bdd2-46bf-b536-d32ae91d3043\") "
Oct 30 11:54:10.006669 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.006625    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:10.008586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.008370    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.009039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.008754    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.009964 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.009934    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:10.015552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.015489    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.016680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.016647    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.016909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.016880    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.020108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.020023    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:10.021878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.021671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.031798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.031760    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz" pod "cdc77535-bdd2-46bf-b536-d32ae91d3043" (UID: "cdc77535-bdd2-46bf-b536-d32ae91d3043"). InnerVolumeSpecName "kube-api-access-ckxdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.036825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.036796    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:54:10.041770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.041740    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:54:10.045489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.044811    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:54:10.101305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101264    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101294    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101307    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101322    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxdz\" (UniqueName: \"kubernetes.io/projected/cdc77535-bdd2-46bf-b536-d32ae91d3043-kube-api-access-ckxdz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101337    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc77535-bdd2-46bf-b536-d32ae91d3043-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101350    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101362    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101374    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.101675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.101386    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc77535-bdd2-46bf-b536-d32ae91d3043-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.202813 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.202779    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.203063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.203043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.203190 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.203177    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.203387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.203372    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.203899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.203537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.204305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.204240    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.204438 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.203840    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.204629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.204614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.205030 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.205005    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.205753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.204939    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.205753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.205296    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.205966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.205948    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.206151 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.206136    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.206867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.206686    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:10.207032 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.206996    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.207263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.207117    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.207381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.207364    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.207811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.207793    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.208308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208277    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.208636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208134    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.208636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208190    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.208636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208226    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.208636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.208976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208945    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.209101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.208980    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "kube-api-access-xz6cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.209101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209017    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.209101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209066    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.209407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209389    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.209529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209432    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.209592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209543    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.209646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209592    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.209703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209656    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.209989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.209953    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.210162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:10.210222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210159    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.210300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210281    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.210564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210441    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.210564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.210768 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210567    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:10.210934 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.210917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.211337 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.211321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.211786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.211676    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.212352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.212196    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:10.212352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.212326    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token" pod "d2248c34-0ee8-4d0a-8b6d-0233e3a29790" (UID: "d2248c34-0ee8-4d0a-8b6d-0233e3a29790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.212569 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.211745    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert\") pod \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\" (UID: \"d2248c34-0ee8-4d0a-8b6d-0233e3a29790\") "
Oct 30 11:54:10.212665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.212613    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.212934 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.212917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo\") pod \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\" (UID: \"af2c3a63-dd7a-43cc-96a2-7186fd13763e\") "
Oct 30 11:54:10.213248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.213094    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md\") pod \"70b61c36-363f-4776-8822-2b6b8cbe04de\" (UID: \"70b61c36-363f-4776-8822-2b6b8cbe04de\") "
Oct 30 11:54:10.214133 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214113    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214279    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214159    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:10.214593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214581    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214691    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.214797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214701    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214744    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214763    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214778    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.214797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214792    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214807    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214821    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz6cf\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-kube-api-access-xz6cf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214836    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214851    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214863    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214878    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214892    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b61c36-363f-4776-8822-2b6b8cbe04de-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214907    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214922    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214936    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b61c36-363f-4776-8822-2b6b8cbe04de-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214950    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214964    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.214979    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2248c34-0ee8-4d0a-8b6d-0233e3a29790-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.215981 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.215947    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:10.217132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.217096    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:10.217625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.217600    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md" pod "70b61c36-363f-4776-8822-2b6b8cbe04de" (UID: "70b61c36-363f-4776-8822-2b6b8cbe04de"). InnerVolumeSpecName "kube-api-access-7g2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.218412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.218389    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m" pod "af2c3a63-dd7a-43cc-96a2-7186fd13763e" (UID: "af2c3a63-dd7a-43cc-96a2-7186fd13763e"). InnerVolumeSpecName "kube-api-access-r2q7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:10.319477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.316989    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.319477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.317034    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b61c36-363f-4776-8822-2b6b8cbe04de-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.319477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.317049    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.319477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.317064    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2md\" (UniqueName: \"kubernetes.io/projected/70b61c36-363f-4776-8822-2b6b8cbe04de-kube-api-access-7g2md\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.319477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.317078    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2q7m\" (UniqueName: \"kubernetes.io/projected/af2c3a63-dd7a-43cc-96a2-7186fd13763e-kube-api-access-r2q7m\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.319477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.317092    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2c3a63-dd7a-43cc-96a2-7186fd13763e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:10.341844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.341818    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"]
Oct 30 11:54:10.342737 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342708    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy"
Oct 30 11:54:10.342737 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342730    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342751    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342758    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342778    5060 cpu_manager.go:414] "containerMap: removing container" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342784    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342799    5060 cpu_manager.go:414] "containerMap: removing container" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342805    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342835    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy"
Oct 30 11:54:10.342927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342842    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy"
Oct 30 11:54:10.343319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.342992    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy"
Oct 30 11:54:10.343319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.343015    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerName="istio-proxy"
Oct 30 11:54:10.343319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.343028    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerName="istio-proxy"
Oct 30 11:54:10.343319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.343040    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" containerName="istio-proxy"
Oct 30 11:54:10.350967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.350938    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.356782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.355981    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"]
Oct 30 11:54:10.367689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.367662    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c8bb87-openshift-default-dockercfg-2d852\""
Oct 30 11:54:10.420207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.420168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:10.451840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.451809    5060 generic.go:356] "Generic (PLEG): container finished" podID="70b61c36-363f-4776-8822-2b6b8cbe04de" containerID="806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68" exitCode=0
Oct 30 11:54:10.452027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.451909    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" event={"ID":"70b61c36-363f-4776-8822-2b6b8cbe04de","Type":"ContainerDied","Data":"806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68"}
Oct 30 11:54:10.452027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.451919    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"
Oct 30 11:54:10.452027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.451967    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t" event={"ID":"70b61c36-363f-4776-8822-2b6b8cbe04de","Type":"ContainerDied","Data":"42014390ad77a11aadaeba75510e58eaa4e211c7f6557eb08264f8c8d3ae8d5f"}
Oct 30 11:54:10.452027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.451992    5060 scope.go:122] "RemoveContainer" containerID="806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68"
Oct 30 11:54:10.454301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.454271    5060 generic.go:356] "Generic (PLEG): container finished" podID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerID="f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3" exitCode=0
Oct 30 11:54:10.454641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.454623    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"
Oct 30 11:54:10.455083 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.455057    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" event={"ID":"d2248c34-0ee8-4d0a-8b6d-0233e3a29790","Type":"ContainerDied","Data":"f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3"}
Oct 30 11:54:10.455163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.455131    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h" event={"ID":"d2248c34-0ee8-4d0a-8b6d-0233e3a29790","Type":"ContainerDied","Data":"34ce250e934efce5910ac99a7815bb78feff437646fc70ab33f89e376783a1e5"}
Oct 30 11:54:10.456930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.456903    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j" event={"ID":"af2c3a63-dd7a-43cc-96a2-7186fd13763e","Type":"ContainerDied","Data":"2d37b528c25102ce628de85ce4f55e6d4fec1b9cfc69212efd0f9a539a8b856f"}
Oct 30 11:54:10.457044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.457024    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"
Oct 30 11:54:10.458660 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.458634    5060 generic.go:356] "Generic (PLEG): container finished" podID="cdc77535-bdd2-46bf-b536-d32ae91d3043" containerID="2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9" exitCode=0
Oct 30 11:54:10.458762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.458684    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"
Oct 30 11:54:10.458762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.458723    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" event={"ID":"cdc77535-bdd2-46bf-b536-d32ae91d3043","Type":"ContainerDied","Data":"2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9"}
Oct 30 11:54:10.458837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.458778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75" event={"ID":"cdc77535-bdd2-46bf-b536-d32ae91d3043","Type":"ContainerDied","Data":"adb5fabcd9959fea550b84595c501b51f859f77acd88ac684d4e98d68d1c4d1a"}
Oct 30 11:54:10.465370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.465350    5060 scope.go:122] "RemoveContainer" containerID="806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68"
Oct 30 11:54:10.465728 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:10.465706    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68\": container with ID starting with 806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68 not found: ID does not exist" containerID="806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68"
Oct 30 11:54:10.465818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.465738    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68"} err="failed to get container status \"806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68\": rpc error: code = NotFound desc = could not find container \"806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68\": container with ID starting with 806d31c759bf698b1a1479e8a2e955e47d2be9920f096bf640de7fed7adfdc68 not found: ID does not exist"
Oct 30 11:54:10.465818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.465758    5060 scope.go:122] "RemoveContainer" containerID="f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3"
Oct 30 11:54:10.477314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.477177    5060 scope.go:122] "RemoveContainer" containerID="9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521381    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scx67\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521485    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521592    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521634    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521745    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521803    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521840    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.523518 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.521963    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.572188    5060 scope.go:122] "RemoveContainer" containerID="f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.576762    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:10.577244    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3\": container with ID starting with f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3 not found: ID does not exist" containerID="f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.577296    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3"} err="failed to get container status \"f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3\": rpc error: code = NotFound desc = could not find container \"f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3\": container with ID starting with f2bcbeb14864816ca3a522cb4ff9865dbb36d92db16cfdf8cfe17f04b9f836f3 not found: ID does not exist"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.577322    5060 scope.go:122] "RemoveContainer" containerID="9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:10.577707    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102\": container with ID starting with 9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102 not found: ID does not exist" containerID="9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.577736    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102"} err="failed to get container status \"9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102\": rpc error: code = NotFound desc = could not find container \"9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102\": container with ID starting with 9a0728be31d353c0396733b2c4e88e410bcea28f32b21af454c8cd73acd34102 not found: ID does not exist"
Oct 30 11:54:10.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.577756    5060 scope.go:122] "RemoveContainer" containerID="c0e6f38c8d0815ae0faec50a9a5883d3986f1a7b9cca45910b2e3d19eada0285"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.619959    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"]
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.627354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630548    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630619    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630819    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630944    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.630997    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.631043    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scx67\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.632523    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.636551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.632838    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.643111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.643078    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.664758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.648883    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.664999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.649975    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.677161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.676004    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scx67\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.683805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.679602    5060 scope.go:122] "RemoveContainer" containerID="2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9"
Oct 30 11:54:10.692352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.692040    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.692352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.692285    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.706212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.706100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo\") pod \"e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") " pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:10.763525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.761668    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7sl2h"]
Oct 30 11:54:10.777232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.776540    5060 scope.go:122] "RemoveContainer" containerID="2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9"
Oct 30 11:54:10.777232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.776656    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"]
Oct 30 11:54:10.777232 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:10.777170    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9\": container with ID starting with 2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9 not found: ID does not exist" containerID="2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9"
Oct 30 11:54:10.777232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.777196    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9"} err="failed to get container status \"2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9\": rpc error: code = NotFound desc = could not find container \"2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9\": container with ID starting with 2d029d0bbb15e8fd156b85d7bf6809c8ecfd5f4e2ce7005d024e18767d35fce9 not found: ID does not exist"
Oct 30 11:54:10.778537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.778514    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mq42j"]
Oct 30 11:54:10.785570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.785506    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"]
Oct 30 11:54:10.792306 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.792231    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zp75"]
Oct 30 11:54:10.799028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.798992    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"]
Oct 30 11:54:10.805743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.805703    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9954t"]
Oct 30 11:54:10.996652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:10.996552    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:11.417321 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:54:11.417275    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66e7a5d_6f51_4d87_a0fe_0dcfba8deefa.slice/crio-eafa5b4ffc0f9f0c87ebb703603371933b43afbcd4f2834edf18777e9fcd30c0 WatchSource:0}: Error finding container eafa5b4ffc0f9f0c87ebb703603371933b43afbcd4f2834edf18777e9fcd30c0: Status 404 returned error can't find the container with id eafa5b4ffc0f9f0c87ebb703603371933b43afbcd4f2834edf18777e9fcd30c0
Oct 30 11:54:11.418440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:11.418400    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"]
Oct 30 11:54:11.421195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:11.421153    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:54:11.421297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:11.421248    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:54:11.421335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:11.421297    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:54:11.467350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:11.467256    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:11.498653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:11.498607    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" event={"ID":"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa","Type":"ContainerStarted","Data":"eafa5b4ffc0f9f0c87ebb703603371933b43afbcd4f2834edf18777e9fcd30c0"}
Oct 30 11:54:12.337971 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.337931    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b61c36-363f-4776-8822-2b6b8cbe04de" path="/var/lib/kubelet/pods/70b61c36-363f-4776-8822-2b6b8cbe04de/volumes"
Oct 30 11:54:12.338489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.338470    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2c3a63-dd7a-43cc-96a2-7186fd13763e" path="/var/lib/kubelet/pods/af2c3a63-dd7a-43cc-96a2-7186fd13763e/volumes"
Oct 30 11:54:12.338903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.338890    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc77535-bdd2-46bf-b536-d32ae91d3043" path="/var/lib/kubelet/pods/cdc77535-bdd2-46bf-b536-d32ae91d3043/volumes"
Oct 30 11:54:12.339705 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.339687    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" path="/var/lib/kubelet/pods/d2248c34-0ee8-4d0a-8b6d-0233e3a29790/volumes"
Oct 30 11:54:12.513921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.513882    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" event={"ID":"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa","Type":"ContainerStarted","Data":"33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"}
Oct 30 11:54:12.587440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.587361    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podStartSLOduration=2.587337712 podStartE2EDuration="2.587337712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:12.572103483 +0000 UTC m=+1456.366987978" watchObservedRunningTime="2026-10-30 11:54:12.587337712 +0000 UTC m=+1456.382222208"
Oct 30 11:54:12.998864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:12.998825    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:13.007212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.007177    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:13.372183 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.372089    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:54:13.384292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.384260    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:54:13.525878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.525849    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:13.527204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.527181    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:54:13.601713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.601679    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:13.745916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:13.745883    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:14.360516 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.359297    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:54:14.466301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.466272    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:54:14.473160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.473127    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:54:14.519605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.519572    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.519790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.519632    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.519790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.519687    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.519790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.519720    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.519790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.519750    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.520026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.519967    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.520026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520010    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.520132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.520132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520106    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.520237 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520147    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.520237 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520186    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.520339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520234    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.520339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520267    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.520339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520312    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.520552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520363    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.520552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520415    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs\") pod \"22894126-e4c1-4cb1-acd1-2819b70eafe5\" (UID: \"22894126-e4c1-4cb1-acd1-2819b70eafe5\") "
Oct 30 11:54:14.520552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520445    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.521058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.520012    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.521861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.521830    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.522179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.522152    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:14.522794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.522759    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.522999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.522978    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "kube-api-access-gf854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.524113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.523781    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.525528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.524266    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526012    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526161    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526186    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/521369ae-35c1-4453-a4ac-f8b19403726a-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526199    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526211    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526222    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf854\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-kube-api-access-gf854\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526240    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526253    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526270    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:14.526766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.526607    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.527595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.527561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.530440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.530413    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.547145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.547107    5060 generic.go:356] "Generic (PLEG): container finished" podID="521369ae-35c1-4453-a4ac-f8b19403726a" containerID="c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a" exitCode=0
Oct 30 11:54:14.547585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.547560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" event={"ID":"521369ae-35c1-4453-a4ac-f8b19403726a","Type":"ContainerDied","Data":"c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a"}
Oct 30 11:54:14.547815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.547797    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f" event={"ID":"521369ae-35c1-4453-a4ac-f8b19403726a","Type":"ContainerDied","Data":"62b974731cc3baf7ef286b627c1d9f6ffc29d68b99b74ceca917bb224287849e"}
Oct 30 11:54:14.547963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.547924    5060 scope.go:122] "RemoveContainer" containerID="c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a"
Oct 30 11:54:14.548298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.548280    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"
Oct 30 11:54:14.553169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.553134    5060 generic.go:356] "Generic (PLEG): container finished" podID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerID="0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f" exitCode=0
Oct 30 11:54:14.553437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.553403    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" event={"ID":"22894126-e4c1-4cb1-acd1-2819b70eafe5","Type":"ContainerDied","Data":"0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f"}
Oct 30 11:54:14.553533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.553509    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v" event={"ID":"22894126-e4c1-4cb1-acd1-2819b70eafe5","Type":"ContainerDied","Data":"57bd338f373a823781363b4443836e425b5756ff2f8520496cd7729a209743fb"}
Oct 30 11:54:14.553685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.553670    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"
Oct 30 11:54:14.556912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.556794    5060 generic.go:356] "Generic (PLEG): container finished" podID="ef4c8b8c-db57-46ce-911c-50372beca916" containerID="4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06" exitCode=0
Oct 30 11:54:14.557342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.557307    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" event={"ID":"ef4c8b8c-db57-46ce-911c-50372beca916","Type":"ContainerDied","Data":"4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06"}
Oct 30 11:54:14.557435 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.557370    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"
Oct 30 11:54:14.557531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.557393    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj" event={"ID":"ef4c8b8c-db57-46ce-911c-50372beca916","Type":"ContainerDied","Data":"7bbafe13736a5a808d8c99b15216c29181ed51218e99202738da0c95ed2fa010"}
Oct 30 11:54:14.557889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.557736    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.557889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.557828    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo" pod "22894126-e4c1-4cb1-acd1-2819b70eafe5" (UID: "22894126-e4c1-4cb1-acd1-2819b70eafe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:14.558066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.558010    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.562235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.561559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.564156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.563253    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:14.575883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.575853    5060 scope.go:122] "RemoveContainer" containerID="c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a"
Oct 30 11:54:14.576279 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:14.576251    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a\": container with ID starting with c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a not found: ID does not exist" containerID="c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a"
Oct 30 11:54:14.576430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.576290    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a"} err="failed to get container status \"c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a\": rpc error: code = NotFound desc = could not find container \"c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a\": container with ID starting with c531dfc86462b3366f52f62be63532ee2e6d629af846292894ab97ab4de6888a not found: ID does not exist"
Oct 30 11:54:14.576430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.576316    5060 scope.go:122] "RemoveContainer" containerID="0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f"
Oct 30 11:54:14.589958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.589926    5060 scope.go:122] "RemoveContainer" containerID="0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f"
Oct 30 11:54:14.590340 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:14.590317    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f\": container with ID starting with 0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f not found: ID does not exist" containerID="0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f"
Oct 30 11:54:14.590430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.590351    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f"} err="failed to get container status \"0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f\": rpc error: code = NotFound desc = could not find container \"0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f\": container with ID starting with 0e7070ca59f50b28014e9e62c973f112c9437977f39abad48d5b2e93c4ea6b1f not found: ID does not exist"
Oct 30 11:54:14.590430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.590376    5060 scope.go:122] "RemoveContainer" containerID="4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06"
Oct 30 11:54:14.605945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.605924    5060 scope.go:122] "RemoveContainer" containerID="4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06"
Oct 30 11:54:14.606328 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:14.606307    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06\": container with ID starting with 4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06 not found: ID does not exist" containerID="4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06"
Oct 30 11:54:14.606370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.606338    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06"} err="failed to get container status \"4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06\": rpc error: code = NotFound desc = could not find container \"4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06\": container with ID starting with 4479294824f13dca4c7ac18c46b08a56e696ae66efd565368e8ae45d65f94d06 not found: ID does not exist"
Oct 30 11:54:14.627592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627555    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.627798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627639    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.627798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627685    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.627798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627710    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.627798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627737    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.628130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627887    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.628130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.627924    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.628130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628054    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.628232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628217    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.628279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628259    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.628360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628343    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.628474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628415    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497\") pod \"521369ae-35c1-4453-a4ac-f8b19403726a\" (UID: \"521369ae-35c1-4453-a4ac-f8b19403726a\") "
Oct 30 11:54:14.628743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628594    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.628743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628672    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs\") pod \"ef4c8b8c-db57-46ce-911c-50372beca916\" (UID: \"ef4c8b8c-db57-46ce-911c-50372beca916\") "
Oct 30 11:54:14.628862 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628667    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.628984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.628876    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.629582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629557    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629704    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629799    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629820 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629821    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629834    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4c8b8c-db57-46ce-911c-50372beca916-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629847    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629860    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22894126-e4c1-4cb1-acd1-2819b70eafe5-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629872    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629884    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629896    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629908    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22894126-e4c1-4cb1-acd1-2819b70eafe5-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629920    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4c8b8c-db57-46ce-911c-50372beca916-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629932    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629947    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629962    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/521369ae-35c1-4453-a4ac-f8b19403726a-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.629980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.629975    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4c8b8c-db57-46ce-911c-50372beca916-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.630524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.630404    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.631341 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.631320    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "kube-api-access-4t497". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.631392 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.631350    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth" pod "ef4c8b8c-db57-46ce-911c-50372beca916" (UID: "ef4c8b8c-db57-46ce-911c-50372beca916"). InnerVolumeSpecName "kube-api-access-9hlth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.632282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.632257    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo" pod "521369ae-35c1-4453-a4ac-f8b19403726a" (UID: "521369ae-35c1-4453-a4ac-f8b19403726a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:14.730956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.730925    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.730956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.730954    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/521369ae-35c1-4453-a4ac-f8b19403726a-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.730956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.730964    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t497\" (UniqueName: \"kubernetes.io/projected/521369ae-35c1-4453-a4ac-f8b19403726a-kube-api-access-4t497\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.731197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.730975    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlth\" (UniqueName: \"kubernetes.io/projected/ef4c8b8c-db57-46ce-911c-50372beca916-kube-api-access-9hlth\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.899944 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.899903    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"]
Oct 30 11:54:14.912571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.912524    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-v8p2f"]
Oct 30 11:54:14.947901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.947867    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"]
Oct 30 11:54:14.952197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.952165    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-7c5gj"]
Oct 30 11:54:14.987050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.987007    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"]
Oct 30 11:54:14.991330 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:14.991298    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-9784v"]
Oct 30 11:54:15.375540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.375498    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:15.520303 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.520271    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:15.523645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.523620    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"]
Oct 30 11:54:15.524208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524195    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy"
Oct 30 11:54:15.524246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524209    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy"
Oct 30 11:54:15.524246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524224    5060 cpu_manager.go:414] "containerMap: removing container" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy"
Oct 30 11:54:15.524246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524229    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy"
Oct 30 11:54:15.524419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524251    5060 cpu_manager.go:414] "containerMap: removing container" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy"
Oct 30 11:54:15.524419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524257    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy"
Oct 30 11:54:15.524419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524360    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" containerName="istio-proxy"
Oct 30 11:54:15.524419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524379    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" containerName="istio-proxy"
Oct 30 11:54:15.524419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524389    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2248c34-0ee8-4d0a-8b6d-0233e3a29790" containerName="istio-proxy"
Oct 30 11:54:15.524419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.524404    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" containerName="istio-proxy"
Oct 30 11:54:15.528031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.528010    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.541748 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.541711    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kct\" (UniqueName: \"kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct\") pod \"payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.541929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.541853    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.543633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.543608    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5da940\""
Oct 30 11:54:15.543880 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.543855    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5da940-dockercfg-8mxvl\""
Oct 30 11:54:15.546036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.546009    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"]
Oct 30 11:54:15.681526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.681427    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.682068 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.680674    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.682594 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.682572    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kct\" (UniqueName: \"kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct\") pod \"payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.705561 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.705521    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"]
Oct 30 11:54:15.738009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.737971    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:15.757368 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.757338    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"]
Oct 30 11:54:15.757716 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.757699    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:15.794415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.788588    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kct\" (UniqueName: \"kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct\") pod \"payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.844122 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.844082    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:15.888024 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.887982    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksmp\" (UniqueName: \"kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp\") pod \"payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:15.888206 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.888055    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:15.892937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.892894    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:15.989899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.989861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksmp\" (UniqueName: \"kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp\") pod \"payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:15.990124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.989957    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:15.990760 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:15.990736    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:16.014743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.014710    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksmp\" (UniqueName: \"kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp\") pod \"payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:16.049056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.048998    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:16.130949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.121716    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:16.201909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.201757    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:16.259848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.259810    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"]
Oct 30 11:54:16.385359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.385193    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22894126-e4c1-4cb1-acd1-2819b70eafe5" path="/var/lib/kubelet/pods/22894126-e4c1-4cb1-acd1-2819b70eafe5/volumes"
Oct 30 11:54:16.385937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.385913    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521369ae-35c1-4453-a4ac-f8b19403726a" path="/var/lib/kubelet/pods/521369ae-35c1-4453-a4ac-f8b19403726a/volumes"
Oct 30 11:54:16.386602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.386580    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4c8b8c-db57-46ce-911c-50372beca916" path="/var/lib/kubelet/pods/ef4c8b8c-db57-46ce-911c-50372beca916/volumes"
Oct 30 11:54:16.567863 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:54:16.567828    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ac0f0c_48ad_451c_a87f_27360b7b523d.slice/crio-a19993e4d4bcb9ee6c9f17bc77d82e96499314df7838bcd1e561975191a75c07 WatchSource:0}: Error finding container a19993e4d4bcb9ee6c9f17bc77d82e96499314df7838bcd1e561975191a75c07: Status 404 returned error can't find the container with id a19993e4d4bcb9ee6c9f17bc77d82e96499314df7838bcd1e561975191a75c07
Oct 30 11:54:16.569491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.569427    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"]
Oct 30 11:54:16.587752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.587712    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8" event={"ID":"d0ac0f0c-48ad-451c-a87f-27360b7b523d","Type":"ContainerStarted","Data":"a19993e4d4bcb9ee6c9f17bc77d82e96499314df7838bcd1e561975191a75c07"}
Oct 30 11:54:16.590365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.590213    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68" event={"ID":"355ce9ab-4c9b-4094-95ff-929e2034cc61","Type":"ContainerStarted","Data":"8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0"}
Oct 30 11:54:16.590365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.590280    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68" event={"ID":"355ce9ab-4c9b-4094-95ff-929e2034cc61","Type":"ContainerStarted","Data":"facd181c349cc24faf316dcf687bf78c6675b6a5f50fb7310dcc0589092e000f"}
Oct 30 11:54:16.590365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.590310    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:16.647739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:16.642103    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68" podStartSLOduration=1.642081366 podStartE2EDuration="1.642081366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:16.635003363 +0000 UTC m=+1460.429887896" watchObservedRunningTime="2026-10-30 11:54:16.642081366 +0000 UTC m=+1460.436965868"
Oct 30 11:54:17.603546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:17.603498    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8" event={"ID":"d0ac0f0c-48ad-451c-a87f-27360b7b523d","Type":"ContainerStarted","Data":"fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623"}
Oct 30 11:54:17.604063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:17.603627    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:17.646793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:17.646737    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8" podStartSLOduration=2.646718954 podStartE2EDuration="2.646718954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:17.645968353 +0000 UTC m=+1461.440852846" watchObservedRunningTime="2026-10-30 11:54:17.646718954 +0000 UTC m=+1461.441603448"
Oct 30 11:54:17.990227 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:17.990188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:18.160469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.160416    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"]
Oct 30 11:54:18.165099 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.165071    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.181156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.181114    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c8bb87-dockercfg-cdsc8\""
Oct 30 11:54:18.181332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.181177    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c8bb87\""
Oct 30 11:54:18.182653 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.182624    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:18.184812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.184782    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"]
Oct 30 11:54:18.223258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.223175    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz2q\" (UniqueName: \"kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q\") pod \"payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.223519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.223445    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.243171 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.243086    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"]
Oct 30 11:54:18.247428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.247409    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.268682 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.268652    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"]
Oct 30 11:54:18.410474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.371827    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz2q\" (UniqueName: \"kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q\") pod \"payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.421510 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.421418    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.426224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.422698    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.437480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.435196    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz2q\" (UniqueName: \"kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q\") pod \"payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.485376 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.485339    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:18.526416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.526376    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsq5\" (UniqueName: \"kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5\") pod \"payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.526693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.526674    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.629308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.629227    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsq5\" (UniqueName: \"kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5\") pod \"payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.629746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.629670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.630989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.630870    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.653724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.653690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsq5\" (UniqueName: \"kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5\") pod \"payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:18.718845 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.718809    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:18.811773 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:54:18.811711    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca2f64c_870b_4bba_936f_ae9c76a3b0de.slice/crio-256fc9487303bfe70762f03c67a734a92e442a74cf31800c5fa8c673a581a487 WatchSource:0}: Error finding container 256fc9487303bfe70762f03c67a734a92e442a74cf31800c5fa8c673a581a487: Status 404 returned error can't find the container with id 256fc9487303bfe70762f03c67a734a92e442a74cf31800c5fa8c673a581a487
Oct 30 11:54:18.814065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.814034    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"]
Oct 30 11:54:18.862952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:18.862919    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:19.183935 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:54:19.183901    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9fdf9d_d305_4e98_988f_8fd4badf33ea.slice/crio-2fe59b0de8c83a14a9f7327e34883d4eb21f3a7ceeff46ad3e4e4b9a05ada171 WatchSource:0}: Error finding container 2fe59b0de8c83a14a9f7327e34883d4eb21f3a7ceeff46ad3e4e4b9a05ada171: Status 404 returned error can't find the container with id 2fe59b0de8c83a14a9f7327e34883d4eb21f3a7ceeff46ad3e4e4b9a05ada171
Oct 30 11:54:19.184878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.184828    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"]
Oct 30 11:54:19.630422 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.630373    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6" event={"ID":"ca9fdf9d-d305-4e98-988f-8fd4badf33ea","Type":"ContainerStarted","Data":"bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60"}
Oct 30 11:54:19.630867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.630441    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6" event={"ID":"ca9fdf9d-d305-4e98-988f-8fd4badf33ea","Type":"ContainerStarted","Data":"2fe59b0de8c83a14a9f7327e34883d4eb21f3a7ceeff46ad3e4e4b9a05ada171"}
Oct 30 11:54:19.630867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.630626    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:19.632277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.632246    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td" event={"ID":"4ca2f64c-870b-4bba-936f-ae9c76a3b0de","Type":"ContainerStarted","Data":"d2230380644f045a2cb831f428f2f2141d229351e4cea9db3bbac6b1d997a68b"}
Oct 30 11:54:19.632381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.632295    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td" event={"ID":"4ca2f64c-870b-4bba-936f-ae9c76a3b0de","Type":"ContainerStarted","Data":"256fc9487303bfe70762f03c67a734a92e442a74cf31800c5fa8c673a581a487"}
Oct 30 11:54:19.632381 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.632331    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:19.674002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.673939    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6" podStartSLOduration=1.6739265749999999 podStartE2EDuration="1.673926575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:19.671054492 +0000 UTC m=+1463.465938985" watchObservedRunningTime="2026-10-30 11:54:19.673926575 +0000 UTC m=+1463.468811096"
Oct 30 11:54:19.711502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:19.711423    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td" podStartSLOduration=1.711409373 podStartE2EDuration="1.711409373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:19.710947193 +0000 UTC m=+1463.505831687" watchObservedRunningTime="2026-10-30 11:54:19.711409373 +0000 UTC m=+1463.506293867"
Oct 30 11:54:19.871900 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:19.871860    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:54:20.741109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:20.741075    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:20.882705 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:20.882663    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:21.113014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:21.112913    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:21.553701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:21.553659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:22.756729 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.756690    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"]
Oct 30 11:54:22.761056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.761031    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:22.780509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.779709    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"]
Oct 30 11:54:22.792755 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.792721    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:54:22.792928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.792877    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:54:22.792928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.792726    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:54:22.793033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.792984    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:54:22.903280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.903237    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:22.903280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.903279    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w628f\" (UniqueName: \"kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:22.903626 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:22.903409    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.006485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.006429    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.006693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.006510    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w628f\" (UniqueName: \"kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.006693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.006576    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.007387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.007222    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.009185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.009160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.030619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.030585    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w628f\" (UniqueName: \"kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f\") pod \"maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.032258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.032237    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"]
Oct 30 11:54:23.036623 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.036594    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc16095-cpztl"]
Oct 30 11:54:23.079270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.079235    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:23.289896 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.289796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:23.392155 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:54:23.392112    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5d1f9e_0396_4bb4_a539_3fa558c31e46.slice/crio-23b7ab9ef3cfebe6bd57b564bd9da59ffde83b22834ef9346ee6de1e79f07374 WatchSource:0}: Error finding container 23b7ab9ef3cfebe6bd57b564bd9da59ffde83b22834ef9346ee6de1e79f07374: Status 404 returned error can't find the container with id 23b7ab9ef3cfebe6bd57b564bd9da59ffde83b22834ef9346ee6de1e79f07374
Oct 30 11:54:23.394681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.394651    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"]
Oct 30 11:54:23.671488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:23.671419    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl" event={"ID":"df5d1f9e-0396-4bb4-a539-3fa558c31e46","Type":"ContainerStarted","Data":"23b7ab9ef3cfebe6bd57b564bd9da59ffde83b22834ef9346ee6de1e79f07374"}
Oct 30 11:54:24.335727 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:24.335693    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb07c9ba-8eae-4fb7-ac57-e565ebcc5716" path="/var/lib/kubelet/pods/bb07c9ba-8eae-4fb7-ac57-e565ebcc5716/volumes"
Oct 30 11:54:24.689953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:24.689858    5060 generic.go:356] "Generic (PLEG): container finished" podID="df5d1f9e-0396-4bb4-a539-3fa558c31e46" containerID="d8bf04acd342aea5c3edfc35a8420af0acea3eee47f1edfb47e68b116ab09ae7" exitCode=0
Oct 30 11:54:24.691699 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:24.691665    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl" event={"ID":"df5d1f9e-0396-4bb4-a539-3fa558c31e46","Type":"ContainerDied","Data":"d8bf04acd342aea5c3edfc35a8420af0acea3eee47f1edfb47e68b116ab09ae7"}
Oct 30 11:54:25.863667 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.863634    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:25.976482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.976369    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca\") pod \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") "
Oct 30 11:54:25.976674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.976495    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f\" (UniqueName: \"kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f\") pod \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") "
Oct 30 11:54:25.976674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.976637    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca\") pod \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\" (UID: \"df5d1f9e-0396-4bb4-a539-3fa558c31e46\") "
Oct 30 11:54:25.976788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.976750    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca" pod "df5d1f9e-0396-4bb4-a539-3fa558c31e46" (UID: "df5d1f9e-0396-4bb4-a539-3fa558c31e46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:25.977194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.977175    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:25.978779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.978752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca" pod "df5d1f9e-0396-4bb4-a539-3fa558c31e46" (UID: "df5d1f9e-0396-4bb4-a539-3fa558c31e46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:54:25.978852 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:25.978826    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f" pod "df5d1f9e-0396-4bb4-a539-3fa558c31e46" (UID: "df5d1f9e-0396-4bb4-a539-3fa558c31e46"). InnerVolumeSpecName "kube-api-access-w628f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:26.078470 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:26.078414    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w628f\" (UniqueName: \"kubernetes.io/projected/df5d1f9e-0396-4bb4-a539-3fa558c31e46-kube-api-access-w628f\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:26.078650 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:26.078485    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5d1f9e-0396-4bb4-a539-3fa558c31e46-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:26.168838 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:26.168802    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:26.711641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:26.711608    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"
Oct 30 11:54:26.711641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:26.711625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl" event={"ID":"df5d1f9e-0396-4bb4-a539-3fa558c31e46","Type":"ContainerDied","Data":"23b7ab9ef3cfebe6bd57b564bd9da59ffde83b22834ef9346ee6de1e79f07374"}
Oct 30 11:54:26.711857 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:26.711682    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b7ab9ef3cfebe6bd57b564bd9da59ffde83b22834ef9346ee6de1e79f07374"
Oct 30 11:54:27.607319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:27.607291    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:54:28.088983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:28.088940    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:28.619995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:28.619966    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:54:29.977251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:29.977215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:30.647778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:30.647746    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:54:30.647958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:30.647822    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:54:30.748304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:30.748243    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:31.839645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:31.839611    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:33.226268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:33.226222    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"]
Oct 30 11:54:33.409560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:33.409521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:33.570126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:33.570038    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:33.716677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:33.716637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:35.002245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:35.002205    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:35.160387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:35.160350    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:36.277126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:36.275290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:38.227290 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.227241    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689" podUID="ca68d5ec-728b-4ced-9497-44a12e1144b2" containerName="payload-processing" containerID="cri-o://64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217" gracePeriod=25
Oct 30 11:54:38.318526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.318489    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:38.472853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.472444    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:38.585270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.585244    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:54:38.730783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.730750    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume\") pod \"ca68d5ec-728b-4ced-9497-44a12e1144b2\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") "
Oct 30 11:54:38.730783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.730787    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp\" (UniqueName: \"kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp\") pod \"ca68d5ec-728b-4ced-9497-44a12e1144b2\" (UID: \"ca68d5ec-728b-4ced-9497-44a12e1144b2\") "
Oct 30 11:54:38.731131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.731110    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume" pod "ca68d5ec-728b-4ced-9497-44a12e1144b2" (UID: "ca68d5ec-728b-4ced-9497-44a12e1144b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:38.733233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.733208    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp" pod "ca68d5ec-728b-4ced-9497-44a12e1144b2" (UID: "ca68d5ec-728b-4ced-9497-44a12e1144b2"). InnerVolumeSpecName "kube-api-access-l6hsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:38.829664 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.829574    5060 generic.go:356] "Generic (PLEG): container finished" podID="ca68d5ec-728b-4ced-9497-44a12e1144b2" containerID="64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217" exitCode=0
Oct 30 11:54:38.829822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.829661    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"
Oct 30 11:54:38.829822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.829665    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689" event={"ID":"ca68d5ec-728b-4ced-9497-44a12e1144b2","Type":"ContainerDied","Data":"64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217"}
Oct 30 11:54:38.829822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.829745    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689" event={"ID":"ca68d5ec-728b-4ced-9497-44a12e1144b2","Type":"ContainerDied","Data":"8448274881a2bf49609084b0f15689dc78224ea52958a1c1e9af48c97c7ec5f8"}
Oct 30 11:54:38.829822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.829777    5060 scope.go:122] "RemoveContainer" containerID="64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217"
Oct 30 11:54:38.832292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.832270    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca68d5ec-728b-4ced-9497-44a12e1144b2-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:38.832419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.832295    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hsp\" (UniqueName: \"kubernetes.io/projected/ca68d5ec-728b-4ced-9497-44a12e1144b2-kube-api-access-l6hsp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:38.841958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.841936    5060 scope.go:122] "RemoveContainer" containerID="64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217"
Oct 30 11:54:38.842271 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:38.842252    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217\": container with ID starting with 64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217 not found: ID does not exist" containerID="64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217"
Oct 30 11:54:38.842334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.842281    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217"} err="failed to get container status \"64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217\": rpc error: code = NotFound desc = could not find container \"64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217\": container with ID starting with 64b663060345456f830205fa94edb89b36aa9f309ed91fa8cc82ef264cc66217 not found: ID does not exist"
Oct 30 11:54:38.884141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.884106    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"]
Oct 30 11:54:38.893470 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:38.893422    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84fed1-5f977c9ccd-m4689"]
Oct 30 11:54:40.336611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:40.336569    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca68d5ec-728b-4ced-9497-44a12e1144b2" path="/var/lib/kubelet/pods/ca68d5ec-728b-4ced-9497-44a12e1144b2/volumes"
Oct 30 11:54:40.548726 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:40.548695    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:42.454489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:42.454420    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:42.599841 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:42.599804    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:42.738684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:42.738577    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:44.556713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:44.556672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:46.255073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:46.255039    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:48.035463 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:48.035426    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:48.908277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:48.908237    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"]
Oct 30 11:54:48.908560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:48.908529    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy" containerID="cri-o://d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d" gracePeriod=30
Oct 30 11:54:49.569476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:49.560923    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:49.857601 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:49.857517    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:54:49.931298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:49.931264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:52.287974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:52.287938    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:52.423990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:52.423951    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:52.614512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:52.614379    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:52.909344 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:52.909254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:53.213783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:53.213747    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:54.287926 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.287900    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:54:54.371046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371015    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371196    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371226    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371366 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371284    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371366 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371322    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371366    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371417    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371471    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.371683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.371660    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket\") pod \"6a1ac9d1-de57-4606-aacb-aaf42086c064\" (UID: \"6a1ac9d1-de57-4606-aacb-aaf42086c064\") "
Oct 30 11:54:54.373006 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.372906    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:54.373222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.373133    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:54.373222 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.373044    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:54.373488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.373428    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:54.373583 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.373522    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:54.375307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.375194    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:54.375405 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.375350    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:54.403474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.400445    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "kube-api-access-vjjlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:54.414028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.413987    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy" pod "6a1ac9d1-de57-4606-aacb-aaf42086c064" (UID: "6a1ac9d1-de57-4606-aacb-aaf42086c064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:54.473826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473783    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjlw\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-kube-api-access-vjjlw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.473826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473824    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473840    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473852    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473866    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473879    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473890    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1ac9d1-de57-4606-aacb-aaf42086c064-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473902    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1ac9d1-de57-4606-aacb-aaf42086c064-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.474057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.473915    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1ac9d1-de57-4606-aacb-aaf42086c064-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:54:54.626498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.626170    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:54.631708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.631658    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"]
Oct 30 11:54:54.635898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.635869    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"]
Oct 30 11:54:54.808542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:54.802035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:55.059535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.058651    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:55.112506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.111032    5060 generic.go:356] "Generic (PLEG): container finished" podID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerID="d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d" exitCode=0
Oct 30 11:54:55.112506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.111103    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" event={"ID":"6a1ac9d1-de57-4606-aacb-aaf42086c064","Type":"ContainerDied","Data":"d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d"}
Oct 30 11:54:55.112506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.111160    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2" event={"ID":"6a1ac9d1-de57-4606-aacb-aaf42086c064","Type":"ContainerDied","Data":"9518420e2db2b85cec5c9cf85817ad451ced82dce9ff65510333a28b5b30808f"}
Oct 30 11:54:55.112506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.111193    5060 scope.go:122] "RemoveContainer" containerID="d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d"
Oct 30 11:54:55.112506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.111403    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"
Oct 30 11:54:55.146618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.146550    5060 scope.go:122] "RemoveContainer" containerID="d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d"
Oct 30 11:54:55.146971 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:54:55.146945    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d\": container with ID starting with d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d not found: ID does not exist" containerID="d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d"
Oct 30 11:54:55.147050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.146978    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d"} err="failed to get container status \"d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d\": rpc error: code = NotFound desc = could not find container \"d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d\": container with ID starting with d30ef0bbd32fc03f812ac76e9e97bf49591c464b0d52358aab5901c687d07a0d not found: ID does not exist"
Oct 30 11:54:55.215478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.214511    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"]
Oct 30 11:54:55.223731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.217830    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-fdvs2"]
Oct 30 11:54:55.239670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:55.234441    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:56.337652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:56.337620    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" path="/var/lib/kubelet/pods/6a1ac9d1-de57-4606-aacb-aaf42086c064/volumes"
Oct 30 11:54:56.624319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:56.624236    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:54:56.630546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:56.630518    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:54:56.719952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:56.719911    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:56.854603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:56.854574    5060 scope.go:122] "RemoveContainer" containerID="f4807a4dfbb8a48904da66516d5db681ea80666698368ba49bf55fb5865791e7"
Oct 30 11:54:56.865300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:56.865279    5060 scope.go:122] "RemoveContainer" containerID="2d248ebccdc29c41c140dd63793087bc38df2b5b5294881e230da6825d95824b"
Oct 30 11:54:57.205096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:57.205060    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:57.347055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:57.347018    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:58.917611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:58.917574    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:59.044523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.044489    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"]
Oct 30 11:54:59.045115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045100    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ca68d5ec-728b-4ced-9497-44a12e1144b2" containerName="payload-processing"
Oct 30 11:54:59.045185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045117    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca68d5ec-728b-4ced-9497-44a12e1144b2" containerName="payload-processing"
Oct 30 11:54:59.045185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045146    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy"
Oct 30 11:54:59.045185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045151    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy"
Oct 30 11:54:59.045291 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045187    5060 cpu_manager.go:414] "containerMap: removing container" podUID="df5d1f9e-0396-4bb4-a539-3fa558c31e46" containerName="revoke-keys"
Oct 30 11:54:59.045291 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045192    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5d1f9e-0396-4bb4-a539-3fa558c31e46" containerName="revoke-keys"
Oct 30 11:54:59.045291 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045284    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5d1f9e-0396-4bb4-a539-3fa558c31e46" containerName="revoke-keys"
Oct 30 11:54:59.045387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045296    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca68d5ec-728b-4ced-9497-44a12e1144b2" containerName="payload-processing"
Oct 30 11:54:59.045387 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.045306    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1ac9d1-de57-4606-aacb-aaf42086c064" containerName="istio-proxy"
Oct 30 11:54:59.051016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.050991    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.068227 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.068194    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"]
Oct 30 11:54:59.073085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.073049    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:59.075742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.075714    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eaee54\""
Oct 30 11:54:59.075742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.075718    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eaee54-dockercfg-zpz82\""
Oct 30 11:54:59.275505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.224305    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.275505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.224617    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8wd\" (UniqueName: \"kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.332480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.327702    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8wd\" (UniqueName: \"kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.343732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.343563    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.344872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.344841    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.368494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.368437    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"]
Oct 30 11:54:59.371900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.371855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:54:59.373959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.373933    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"]
Oct 30 11:54:59.374098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.374082    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.375058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.375032    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8wd\" (UniqueName: \"kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.546920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.546826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.547086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.547008    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqlgq\" (UniqueName: \"kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq\") pod \"payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.632630 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.632573    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b" podUID="fe0b62cc-764b-41b5-96c7-70d9832e6fe0" containerName="payload-processing" containerID="cri-o://ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a" gracePeriod=25
Oct 30 11:54:59.637161 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.637119    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296" podUID="5f9f1ab8-0bb2-42a8-9013-f871ac89900a" containerName="payload-pre-processing" containerID="cri-o://22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc" gracePeriod=25
Oct 30 11:54:59.648345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.648309    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqlgq\" (UniqueName: \"kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq\") pod \"payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.648531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.648510    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.649241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.649216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.666664 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.666638    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:54:59.672236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.672204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqlgq\" (UniqueName: \"kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq\") pod \"payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:54:59.739720 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:54:59.739685    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:55:00.017067 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.017013    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"]
Oct 30 11:55:00.054997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.054961    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:55:00.121869 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.121840    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:55:00.155342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.155308    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh\" (UniqueName: \"kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh\") pod \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") "
Oct 30 11:55:00.155616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.155600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume\") pod \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\" (UID: \"fe0b62cc-764b-41b5-96c7-70d9832e6fe0\") "
Oct 30 11:55:00.156025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.155998    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume" pod "fe0b62cc-764b-41b5-96c7-70d9832e6fe0" (UID: "fe0b62cc-764b-41b5-96c7-70d9832e6fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:00.157621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.157599    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh" pod "fe0b62cc-764b-41b5-96c7-70d9832e6fe0" (UID: "fe0b62cc-764b-41b5-96c7-70d9832e6fe0"). InnerVolumeSpecName "kube-api-access-v4tjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:00.196671 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:00.196601    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4c51be_cb35_4bbd_a0aa_0651a5aa06e4.slice/crio-0e96f60d89a96a0c55195e46b9534db1ee1c0b6a93d95de7f95e735f826616b5 WatchSource:0}: Error finding container 0e96f60d89a96a0c55195e46b9534db1ee1c0b6a93d95de7f95e735f826616b5: Status 404 returned error can't find the container with id 0e96f60d89a96a0c55195e46b9534db1ee1c0b6a93d95de7f95e735f826616b5
Oct 30 11:55:00.203111 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.202612    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"]
Oct 30 11:55:00.257115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.257084    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz\" (UniqueName: \"kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz\") pod \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") "
Oct 30 11:55:00.257252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.257224    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume\") pod \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\" (UID: \"5f9f1ab8-0bb2-42a8-9013-f871ac89900a\") "
Oct 30 11:55:00.257586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.257569    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:00.257586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.257585    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4tjh\" (UniqueName: \"kubernetes.io/projected/fe0b62cc-764b-41b5-96c7-70d9832e6fe0-kube-api-access-v4tjh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:00.257904 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.257619    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume" pod "5f9f1ab8-0bb2-42a8-9013-f871ac89900a" (UID: "5f9f1ab8-0bb2-42a8-9013-f871ac89900a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:00.259296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.259268    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz" pod "5f9f1ab8-0bb2-42a8-9013-f871ac89900a" (UID: "5f9f1ab8-0bb2-42a8-9013-f871ac89900a"). InnerVolumeSpecName "kube-api-access-hc2hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:00.359296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.359260    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2hz\" (UniqueName: \"kubernetes.io/projected/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-kube-api-access-hc2hz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:00.359296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.359294    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9f1ab8-0bb2-42a8-9013-f871ac89900a-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:00.386433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.386396    5060 generic.go:356] "Generic (PLEG): container finished" podID="5f9f1ab8-0bb2-42a8-9013-f871ac89900a" containerID="22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc" exitCode=0
Oct 30 11:55:00.386637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.386503    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"
Oct 30 11:55:00.386637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.386626    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296" event={"ID":"5f9f1ab8-0bb2-42a8-9013-f871ac89900a","Type":"ContainerDied","Data":"22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc"}
Oct 30 11:55:00.386756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.386694    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296" event={"ID":"5f9f1ab8-0bb2-42a8-9013-f871ac89900a","Type":"ContainerDied","Data":"6b85d48dabfc908ac2b9aa5f8b8a5486b4aa9c4c693c0d3a4cffe3f5d092430f"}
Oct 30 11:55:00.386756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.386724    5060 scope.go:122] "RemoveContainer" containerID="22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc"
Oct 30 11:55:00.390607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.389689    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj" event={"ID":"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4","Type":"ContainerStarted","Data":"8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49"}
Oct 30 11:55:00.390607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.389759    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj" event={"ID":"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4","Type":"ContainerStarted","Data":"0e96f60d89a96a0c55195e46b9534db1ee1c0b6a93d95de7f95e735f826616b5"}
Oct 30 11:55:00.390607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.389908    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:55:00.392340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.392317    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj" event={"ID":"ab6c77d4-9990-48ea-82f7-7b65735b346e","Type":"ContainerStarted","Data":"1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8"}
Oct 30 11:55:00.392487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.392358    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj" event={"ID":"ab6c77d4-9990-48ea-82f7-7b65735b346e","Type":"ContainerStarted","Data":"bd576246ddaef1106330d55776fa929804b63e213c8bd802299b55b90957ae33"}
Oct 30 11:55:00.393028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.393011    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:55:00.394868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.394805    5060 generic.go:356] "Generic (PLEG): container finished" podID="fe0b62cc-764b-41b5-96c7-70d9832e6fe0" containerID="ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a" exitCode=0
Oct 30 11:55:00.394991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.394877    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"
Oct 30 11:55:00.394991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.394932    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b" event={"ID":"fe0b62cc-764b-41b5-96c7-70d9832e6fe0","Type":"ContainerDied","Data":"ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a"}
Oct 30 11:55:00.394991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.394970    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b" event={"ID":"fe0b62cc-764b-41b5-96c7-70d9832e6fe0","Type":"ContainerDied","Data":"df4db07cc9e1a0cef9503832a6af502e9a0f79dea4653ea8ff2aeda7106b5717"}
Oct 30 11:55:00.407277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.407235    5060 scope.go:122] "RemoveContainer" containerID="22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc"
Oct 30 11:55:00.407667 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:00.407642    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc\": container with ID starting with 22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc not found: ID does not exist" containerID="22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc"
Oct 30 11:55:00.407779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.407680    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc"} err="failed to get container status \"22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc\": rpc error: code = NotFound desc = could not find container \"22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc\": container with ID starting with 22b17b28e8a88452044b70328f20397c06f7be3db29db47eb8ac62db7523d4dc not found: ID does not exist"
Oct 30 11:55:00.407779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.407704    5060 scope.go:122] "RemoveContainer" containerID="ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a"
Oct 30 11:55:00.418634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.418608    5060 scope.go:122] "RemoveContainer" containerID="ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a"
Oct 30 11:55:00.419005 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:00.418983    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a\": container with ID starting with ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a not found: ID does not exist" containerID="ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a"
Oct 30 11:55:00.419103 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.419015    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a"} err="failed to get container status \"ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a\": rpc error: code = NotFound desc = could not find container \"ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a\": container with ID starting with ca93d2882134288a37c62f372b0cea13db76509db525a941a81bb18dbd65c70a not found: ID does not exist"
Oct 30 11:55:00.446496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.446425    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj" podStartSLOduration=1.44641316 podStartE2EDuration="1.44641316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:00.445860849 +0000 UTC m=+1504.240745345" watchObservedRunningTime="2026-10-30 11:55:00.44641316 +0000 UTC m=+1504.241297653"
Oct 30 11:55:00.488014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.487979    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"]
Oct 30 11:55:00.491751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.491715    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-785c7f6bc9-rw296"]
Oct 30 11:55:00.528782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.528748    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"]
Oct 30 11:55:00.532601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.532574    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-569cdb57b8-6k75b"]
Oct 30 11:55:00.570293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:00.570242    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj" podStartSLOduration=1.57022336 podStartE2EDuration="1.57022336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:00.569208857 +0000 UTC m=+1504.364093351" watchObservedRunningTime="2026-10-30 11:55:00.57022336 +0000 UTC m=+1504.365107855"
Oct 30 11:55:01.727534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:01.727498    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:02.335886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:02.335854    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9f1ab8-0bb2-42a8-9013-f871ac89900a" path="/var/lib/kubelet/pods/5f9f1ab8-0bb2-42a8-9013-f871ac89900a/volumes"
Oct 30 11:55:02.336293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:02.336280    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0b62cc-764b-41b5-96c7-70d9832e6fe0" path="/var/lib/kubelet/pods/fe0b62cc-764b-41b5-96c7-70d9832e6fe0/volumes"
Oct 30 11:55:03.661939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:03.661896    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:04.442003 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:04.441956    5060 generic.go:356] "Generic (PLEG): container finished" podID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerID="33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484" exitCode=0
Oct 30 11:55:04.442298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:04.442019    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" event={"ID":"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa","Type":"ContainerDied","Data":"33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"}
Oct 30 11:55:04.442541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:04.442523    5060 scope.go:122] "RemoveContainer" containerID="33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"
Oct 30 11:55:04.444506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:04.444439    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:04.444604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:04.444568    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:04.444657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:04.444624    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:05.451974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:05.451933    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" event={"ID":"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa","Type":"ContainerStarted","Data":"1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6"}
Oct 30 11:55:05.549260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:05.549229    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:05.999480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:05.998596    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:55:05.999480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:05.998724    5060 patch_prober.go:29] interesting pod/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 11:55:05.999480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:05.998782    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 11:55:07.001483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:06.999925    5060 patch_prober.go:29] interesting pod/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 11:55:07.001483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:06.999993    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 11:55:07.743346 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:07.743317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:07.999540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:07.999347    5060 patch_prober.go:29] interesting pod/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 11:55:07.999540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:07.999407    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 11:55:09.000907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:09.000868    5060 patch_prober.go:29] interesting pod/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 11:55:09.001553 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:09.001515    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 11:55:09.603396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:09.603360    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:09.952436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:09.952398    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"]
Oct 30 11:55:09.973619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:09.973580    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"]
Oct 30 11:55:10.000476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:10.000260    5060 patch_prober.go:29] interesting pod/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 11:55:10.000476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:10.000317    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 11:55:10.123219 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:10.123181    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:10.296546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:10.292976    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:11.097108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:11.097065    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:55:11.097665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:11.097639    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:55:11.098665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:11.098640    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:55:11.414386 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:11.414305    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:55:11.696794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:11.696764    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:12.423786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:12.423756    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:55:13.610244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:13.610197    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:14.876064 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:14.876033    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:14.958615 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:14.958552    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8" podUID="d0ac0f0c-48ad-451c-a87f-27360b7b523d" containerName="payload-processing" containerID="cri-o://fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623" gracePeriod=25
Oct 30 11:55:15.023157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:14.981271    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68" podUID="355ce9ab-4c9b-4094-95ff-929e2034cc61" containerName="payload-pre-processing" containerID="cri-o://8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0" gracePeriod=25
Oct 30 11:55:15.048112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.048061    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"]
Oct 30 11:55:15.072254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.069253    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5d2cc6-7jdwg"]
Oct 30 11:55:15.072254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.071871    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:15.540983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.540957    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:55:15.556065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.556040    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:55:15.576310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.576279    5060 generic.go:356] "Generic (PLEG): container finished" podID="d0ac0f0c-48ad-451c-a87f-27360b7b523d" containerID="fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623" exitCode=0
Oct 30 11:55:15.576310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.576310    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"
Oct 30 11:55:15.576598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.576350    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8" event={"ID":"d0ac0f0c-48ad-451c-a87f-27360b7b523d","Type":"ContainerDied","Data":"fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623"}
Oct 30 11:55:15.576598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.576409    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8" event={"ID":"d0ac0f0c-48ad-451c-a87f-27360b7b523d","Type":"ContainerDied","Data":"a19993e4d4bcb9ee6c9f17bc77d82e96499314df7838bcd1e561975191a75c07"}
Oct 30 11:55:15.576598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.576434    5060 scope.go:122] "RemoveContainer" containerID="fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623"
Oct 30 11:55:15.578797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.578773    5060 generic.go:356] "Generic (PLEG): container finished" podID="355ce9ab-4c9b-4094-95ff-929e2034cc61" containerID="8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0" exitCode=0
Oct 30 11:55:15.578911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.578841    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68" event={"ID":"355ce9ab-4c9b-4094-95ff-929e2034cc61","Type":"ContainerDied","Data":"8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0"}
Oct 30 11:55:15.578911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.578883    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68" event={"ID":"355ce9ab-4c9b-4094-95ff-929e2034cc61","Type":"ContainerDied","Data":"facd181c349cc24faf316dcf687bf78c6675b6a5f50fb7310dcc0589092e000f"}
Oct 30 11:55:15.579193 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.578937    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"
Oct 30 11:55:15.618569 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.618538    5060 scope.go:122] "RemoveContainer" containerID="fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623"
Oct 30 11:55:15.618896 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:15.618874    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623\": container with ID starting with fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623 not found: ID does not exist" containerID="fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623"
Oct 30 11:55:15.618986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.618909    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623"} err="failed to get container status \"fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623\": rpc error: code = NotFound desc = could not find container \"fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623\": container with ID starting with fe8e6b6730bc4eca91a4d311a8115d2373444657624e2f61ec2772b2add8c623 not found: ID does not exist"
Oct 30 11:55:15.618986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.618935    5060 scope.go:122] "RemoveContainer" containerID="8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0"
Oct 30 11:55:15.632593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.632567    5060 scope.go:122] "RemoveContainer" containerID="8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0"
Oct 30 11:55:15.632943 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:15.632923    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0\": container with ID starting with 8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0 not found: ID does not exist" containerID="8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0"
Oct 30 11:55:15.633013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.632952    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0"} err="failed to get container status \"8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0\": rpc error: code = NotFound desc = could not find container \"8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0\": container with ID starting with 8012bed562e454326ecab7c81dc848e5456207342be34a2c298a0707982117c0 not found: ID does not exist"
Oct 30 11:55:15.689906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.689872    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume\") pod \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") "
Oct 30 11:55:15.690159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.689965    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp\" (UniqueName: \"kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp\") pod \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\" (UID: \"d0ac0f0c-48ad-451c-a87f-27360b7b523d\") "
Oct 30 11:55:15.690159 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.690118    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct\" (UniqueName: \"kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct\") pod \"355ce9ab-4c9b-4094-95ff-929e2034cc61\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") "
Oct 30 11:55:15.690290 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.690175    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume\") pod \"355ce9ab-4c9b-4094-95ff-929e2034cc61\" (UID: \"355ce9ab-4c9b-4094-95ff-929e2034cc61\") "
Oct 30 11:55:15.690369 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.690333    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume" pod "d0ac0f0c-48ad-451c-a87f-27360b7b523d" (UID: "d0ac0f0c-48ad-451c-a87f-27360b7b523d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:15.690800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.690774    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac0f0c-48ad-451c-a87f-27360b7b523d-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:15.690923 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.690836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume" pod "355ce9ab-4c9b-4094-95ff-929e2034cc61" (UID: "355ce9ab-4c9b-4094-95ff-929e2034cc61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:15.692936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.692903    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp" pod "d0ac0f0c-48ad-451c-a87f-27360b7b523d" (UID: "d0ac0f0c-48ad-451c-a87f-27360b7b523d"). InnerVolumeSpecName "kube-api-access-rksmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:15.693063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.692977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct" pod "355ce9ab-4c9b-4094-95ff-929e2034cc61" (UID: "355ce9ab-4c9b-4094-95ff-929e2034cc61"). InnerVolumeSpecName "kube-api-access-k7kct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:15.792586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.792493    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce9ab-4c9b-4094-95ff-929e2034cc61-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:15.792586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.792523    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rksmp\" (UniqueName: \"kubernetes.io/projected/d0ac0f0c-48ad-451c-a87f-27360b7b523d-kube-api-access-rksmp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:15.792586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.792533    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kct\" (UniqueName: \"kubernetes.io/projected/355ce9ab-4c9b-4094-95ff-929e2034cc61-kube-api-access-k7kct\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:15.926771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.926733    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"]
Oct 30 11:55:15.933594 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.933564    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-59d67579bd-bjkq8"]
Oct 30 11:55:15.968734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.968699    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"]
Oct 30 11:55:15.980431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:15.980385    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-657ff65d87-vbg68"]
Oct 30 11:55:16.186303 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:16.186198    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:16.335258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:16.335224    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355ce9ab-4c9b-4094-95ff-929e2034cc61" path="/var/lib/kubelet/pods/355ce9ab-4c9b-4094-95ff-929e2034cc61/volumes"
Oct 30 11:55:16.335673 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:16.335658    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ac0f0c-48ad-451c-a87f-27360b7b523d" path="/var/lib/kubelet/pods/d0ac0f0c-48ad-451c-a87f-27360b7b523d/volumes"
Oct 30 11:55:16.336101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:16.336087    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cc7563-e46e-4ca4-81f5-276b739b3b5d" path="/var/lib/kubelet/pods/e0cc7563-e46e-4ca4-81f5-276b739b3b5d/volumes"
Oct 30 11:55:18.322383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:18.322338    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:18.990985 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:18.990944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:19.141974 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:19.141935    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:19.150685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:19.150645    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"]
Oct 30 11:55:19.164003 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:19.163969    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84fed1-qs6dl"]
Oct 30 11:55:19.292287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:19.292196    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:19.674341 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:19.674246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:19.860185 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:19.860142    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:55:20.336360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:20.336316    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5d1f9e-0396-4bb4-a539-3fa558c31e46" path="/var/lib/kubelet/pods/df5d1f9e-0396-4bb4-a539-3fa558c31e46/volumes"
Oct 30 11:55:21.151765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:21.151729    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:21.295723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:21.295683    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:21.954588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:21.954551    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:22.353257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.353168    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"]
Oct 30 11:55:22.354109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354085    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fe0b62cc-764b-41b5-96c7-70d9832e6fe0" containerName="payload-processing"
Oct 30 11:55:22.354109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354110    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0b62cc-764b-41b5-96c7-70d9832e6fe0" containerName="payload-processing"
Oct 30 11:55:22.354292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354125    5060 cpu_manager.go:414] "containerMap: removing container" podUID="355ce9ab-4c9b-4094-95ff-929e2034cc61" containerName="payload-pre-processing"
Oct 30 11:55:22.354292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354133    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355ce9ab-4c9b-4094-95ff-929e2034cc61" containerName="payload-pre-processing"
Oct 30 11:55:22.354292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354149    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f9f1ab8-0bb2-42a8-9013-f871ac89900a" containerName="payload-pre-processing"
Oct 30 11:55:22.354292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354156    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9f1ab8-0bb2-42a8-9013-f871ac89900a" containerName="payload-pre-processing"
Oct 30 11:55:22.354292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354182    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d0ac0f0c-48ad-451c-a87f-27360b7b523d" containerName="payload-processing"
Oct 30 11:55:22.354292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354189    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ac0f0c-48ad-451c-a87f-27360b7b523d" containerName="payload-processing"
Oct 30 11:55:22.354602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354354    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ac0f0c-48ad-451c-a87f-27360b7b523d" containerName="payload-processing"
Oct 30 11:55:22.354602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354379    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0b62cc-764b-41b5-96c7-70d9832e6fe0" containerName="payload-processing"
Oct 30 11:55:22.354602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354389    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="355ce9ab-4c9b-4094-95ff-929e2034cc61" containerName="payload-pre-processing"
Oct 30 11:55:22.354602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.354398    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9f1ab8-0bb2-42a8-9013-f871ac89900a" containerName="payload-pre-processing"
Oct 30 11:55:22.360235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.360208    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.396116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.396084    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"]
Oct 30 11:55:22.396399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.396381    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-73dffe-openshift-default-dockercfg-9glcg\""
Oct 30 11:55:22.414272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414229    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414321    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414351    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414388    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414496    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414712    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414752    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjc4\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414870 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414828    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.414928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.414869    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.453983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.453940    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:22.517992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.517947    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.518745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.518627    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.518956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.518936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.519850 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.519288    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.520040 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.520023    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.533765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.533732    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.534472 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.534433    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.535102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.535082    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjc4\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.535280 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.535261    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.535469 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.535434    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.536938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.520367    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.536938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.534377    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.536938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.535810    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.536938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.536396    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.541471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.538263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.541471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.540723    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.560493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.554411    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.608991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.608674    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjc4\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4\") pod \"e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") " pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:22.670329 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.666446    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:22.695473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:22.694517    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:23.032800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.032761    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"]
Oct 30 11:55:23.044767 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.044715    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cffafa-w58dr"]
Oct 30 11:55:23.057486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.057434    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"]
Oct 30 11:55:23.058618 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:23.058582    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c7d92f_8eef_4be9_b3d7_e0e877130074.slice/crio-6d64537a5f79e00895ac5fd5a118573b971b09741ac5ef7e1c58511639369227 WatchSource:0}: Error finding container 6d64537a5f79e00895ac5fd5a118573b971b09741ac5ef7e1c58511639369227: Status 404 returned error can't find the container with id 6d64537a5f79e00895ac5fd5a118573b971b09741ac5ef7e1c58511639369227
Oct 30 11:55:23.061498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.061467    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:23.061588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.061535    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:23.061588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.061562    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:23.105629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.105580    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:23.249592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.247304    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:23.393061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.392947    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:23.676245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.676145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" event={"ID":"53c7d92f-8eef-4be9-b3d7-e0e877130074","Type":"ContainerStarted","Data":"bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"}
Oct 30 11:55:23.676245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.676215    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" event={"ID":"53c7d92f-8eef-4be9-b3d7-e0e877130074","Type":"ContainerStarted","Data":"6d64537a5f79e00895ac5fd5a118573b971b09741ac5ef7e1c58511639369227"}
Oct 30 11:55:23.695642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.695604    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:23.740371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:23.740301    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podStartSLOduration=1.740277414 podStartE2EDuration="1.740277414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:23.735382972 +0000 UTC m=+1527.530267467" watchObservedRunningTime="2026-10-30 11:55:23.740277414 +0000 UTC m=+1527.535161909"
Oct 30 11:55:24.342476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:24.339423    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a5fe67-da11-4387-a313-e9f0391328f4" path="/var/lib/kubelet/pods/19a5fe67-da11-4387-a313-e9f0391328f4/volumes"
Oct 30 11:55:24.704616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:24.704583    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:25.058991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:25.058897    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:25.708754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:25.708720    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:25.709956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:25.709934    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:55:26.035382 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.035272    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"]
Oct 30 11:55:26.039355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.039325    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b3ec50-rk6ss"]
Oct 30 11:55:26.304677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.304582    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:26.336794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.336762    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8183a93a-076c-4dc1-9daf-30ee1c49733b" path="/var/lib/kubelet/pods/8183a93a-076c-4dc1-9daf-30ee1c49733b/volumes"
Oct 30 11:55:26.490541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.485026    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:26.647287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.645623    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"]
Oct 30 11:55:26.647841 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.647790    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" containerID="cri-o://094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a" gracePeriod=30
Oct 30 11:55:26.702281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:26.702233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:27.326133 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.326102    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"]
Oct 30 11:55:27.331938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.331905    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.348639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.348604    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"]
Oct 30 11:55:27.356889 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.356856    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:55:27.357138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.356900    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:55:27.423756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.423712    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:55:27.423945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.423815    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:55:27.439703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.439667    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.439881 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.439837    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.440076 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.440057    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp44q\" (UniqueName: \"kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.464506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.464468    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:27.541545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.541510    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp44q\" (UniqueName: \"kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.541773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.541576    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.541773 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.541651    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.542342 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.542319    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.544241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.544216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.565575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.565533    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp44q\" (UniqueName: \"kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q\") pod \"maas-api-revoke-keys-e2e-api-b-5389d2-8bm54\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.625540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.625442    5060 patch_prober.go:29] interesting pod/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:55:27.625540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.625527    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:55:27.647323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.647291    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:27.968850 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:27.968816    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49ee66d_6660_4320_ab97_54c3165b0ea6.slice/crio-7523359cb698678cb94ce596b62577820cff3f5ee3a35212a8764ff723237064 WatchSource:0}: Error finding container 7523359cb698678cb94ce596b62577820cff3f5ee3a35212a8764ff723237064: Status 404 returned error can't find the container with id 7523359cb698678cb94ce596b62577820cff3f5ee3a35212a8764ff723237064
Oct 30 11:55:27.969722 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:27.969694    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"]
Oct 30 11:55:28.118560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.118521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:28.204428 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.204392    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"]
Oct 30 11:55:28.210335 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.210309    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.225840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.225807    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-73dffe-dockercfg-h88wv\""
Oct 30 11:55:28.227372 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.227339    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"]
Oct 30 11:55:28.234148 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.234116    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-73dffe\""
Oct 30 11:55:28.275206 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.275174    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"]
Oct 30 11:55:28.282505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.277289    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:28.282505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.279824    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.298610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.298440    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"]
Oct 30 11:55:28.412534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.395859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.412534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.395970    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") " pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.412534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.396082    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxg96\" (UniqueName: \"kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96\") pod \"payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.412534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.396126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wdx\" (UniqueName: \"kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx\") pod \"payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") " pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.498932    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") " pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.499066    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxg96\" (UniqueName: \"kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96\") pod \"payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.499113    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wdx\" (UniqueName: \"kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx\") pod \"payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") " pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.499278    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.500186    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.503162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.502914    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") " pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.527156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.527123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wdx\" (UniqueName: \"kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx\") pod \"payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") " pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.527377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.527351    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxg96\" (UniqueName: \"kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96\") pod \"payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.597328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.597292    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:28.741688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.740995    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54" event={"ID":"c49ee66d-6660-4320-ab97-54c3165b0ea6","Type":"ContainerStarted","Data":"aa5deea568d44c8c0c4cfdc015991f3b3c73ec0fd733f8b1d00ea014f19cf439"}
Oct 30 11:55:28.741688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.741054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54" event={"ID":"c49ee66d-6660-4320-ab97-54c3165b0ea6","Type":"ContainerStarted","Data":"7523359cb698678cb94ce596b62577820cff3f5ee3a35212a8764ff723237064"}
Oct 30 11:55:28.794077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.793954    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54" podStartSLOduration=1.793936932 podStartE2EDuration="1.793936932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:28.791237226 +0000 UTC m=+1532.586121719" watchObservedRunningTime="2026-10-30 11:55:28.793936932 +0000 UTC m=+1532.588821426"
Oct 30 11:55:28.826634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.826595    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:28.922920 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:28.922886    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"]
Oct 30 11:55:28.925781 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:28.925758    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0a9518_4c8c_47f1_94b9_716cdb26004f.slice/crio-b9659c23b36db16787ce11358956f852176d82733bf136d60ea2ac2921756859 WatchSource:0}: Error finding container b9659c23b36db16787ce11358956f852176d82733bf136d60ea2ac2921756859: Status 404 returned error can't find the container with id b9659c23b36db16787ce11358956f852176d82733bf136d60ea2ac2921756859
Oct 30 11:55:29.142759 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.142725    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"]
Oct 30 11:55:29.164911 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:29.164877    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cb61db_69bd_4634_b900_aca2c172aa47.slice/crio-288d484ee8dafaa35ddb74cd14af09f3aebbba958dbe3f3116a9faa5f984b371 WatchSource:0}: Error finding container 288d484ee8dafaa35ddb74cd14af09f3aebbba958dbe3f3116a9faa5f984b371: Status 404 returned error can't find the container with id 288d484ee8dafaa35ddb74cd14af09f3aebbba958dbe3f3116a9faa5f984b371
Oct 30 11:55:29.750772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.750734    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl" event={"ID":"3c0a9518-4c8c-47f1-94b9-716cdb26004f","Type":"ContainerStarted","Data":"eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3"}
Oct 30 11:55:29.751265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.750792    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl" event={"ID":"3c0a9518-4c8c-47f1-94b9-716cdb26004f","Type":"ContainerStarted","Data":"b9659c23b36db16787ce11358956f852176d82733bf136d60ea2ac2921756859"}
Oct 30 11:55:29.751265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.750827    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:29.752528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.752507    5060 generic.go:356] "Generic (PLEG): container finished" podID="c49ee66d-6660-4320-ab97-54c3165b0ea6" containerID="aa5deea568d44c8c0c4cfdc015991f3b3c73ec0fd733f8b1d00ea014f19cf439" exitCode=0
Oct 30 11:55:29.752655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.752582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54" event={"ID":"c49ee66d-6660-4320-ab97-54c3165b0ea6","Type":"ContainerDied","Data":"aa5deea568d44c8c0c4cfdc015991f3b3c73ec0fd733f8b1d00ea014f19cf439"}
Oct 30 11:55:29.754081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.754061    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5" event={"ID":"97cb61db-69bd-4634-b900-aca2c172aa47","Type":"ContainerStarted","Data":"883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed"}
Oct 30 11:55:29.754232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.754095    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5" event={"ID":"97cb61db-69bd-4634-b900-aca2c172aa47","Type":"ContainerStarted","Data":"288d484ee8dafaa35ddb74cd14af09f3aebbba958dbe3f3116a9faa5f984b371"}
Oct 30 11:55:29.754232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.754143    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:29.796541 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.796484    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl" podStartSLOduration=1.796469189 podStartE2EDuration="1.796469189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:29.794218639 +0000 UTC m=+1533.589103130" watchObservedRunningTime="2026-10-30 11:55:29.796469189 +0000 UTC m=+1533.591353677"
Oct 30 11:55:29.887626 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:29.885094    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5" podStartSLOduration=1.8850721209999999 podStartE2EDuration="1.885072121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:29.874856813 +0000 UTC m=+1533.669741308" watchObservedRunningTime="2026-10-30 11:55:29.885072121 +0000 UTC m=+1533.679956615"
Oct 30 11:55:30.048882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:30.048848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:30.946656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:30.946628    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:31.043248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.043154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca\") pod \"c49ee66d-6660-4320-ab97-54c3165b0ea6\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") "
Oct 30 11:55:31.043414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.043250    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q\" (UniqueName: \"kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q\") pod \"c49ee66d-6660-4320-ab97-54c3165b0ea6\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") "
Oct 30 11:55:31.043414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.043299    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca\") pod \"c49ee66d-6660-4320-ab97-54c3165b0ea6\" (UID: \"c49ee66d-6660-4320-ab97-54c3165b0ea6\") "
Oct 30 11:55:31.043778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.043752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca" pod "c49ee66d-6660-4320-ab97-54c3165b0ea6" (UID: "c49ee66d-6660-4320-ab97-54c3165b0ea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:31.043965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.043948    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:31.046287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.046252    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca" pod "c49ee66d-6660-4320-ab97-54c3165b0ea6" (UID: "c49ee66d-6660-4320-ab97-54c3165b0ea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:55:31.046419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.046306    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q" pod "c49ee66d-6660-4320-ab97-54c3165b0ea6" (UID: "c49ee66d-6660-4320-ab97-54c3165b0ea6"). InnerVolumeSpecName "kube-api-access-tp44q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:31.145805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.145770    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49ee66d-6660-4320-ab97-54c3165b0ea6-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:31.145805 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.145803    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp44q\" (UniqueName: \"kubernetes.io/projected/c49ee66d-6660-4320-ab97-54c3165b0ea6-kube-api-access-tp44q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:31.770379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.770334    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54" event={"ID":"c49ee66d-6660-4320-ab97-54c3165b0ea6","Type":"ContainerDied","Data":"7523359cb698678cb94ce596b62577820cff3f5ee3a35212a8764ff723237064"}
Oct 30 11:55:31.770379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.770367    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"
Oct 30 11:55:31.770610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.770395    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7523359cb698678cb94ce596b62577820cff3f5ee3a35212a8764ff723237064"
Oct 30 11:55:31.948795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.948726    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:31.995484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:31.995437    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:55:32.058645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058607    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.058848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058685    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.058848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058743    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.058848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058805    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.058848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058840    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.059086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058881    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.059086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058968    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:55:32.059086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.058983    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.059086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059028    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.059086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:55:32.059353 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059155    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy\") pod \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\" (UID: \"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76\") "
Oct 30 11:55:32.059692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059645    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:55:32.059830 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:55:32.059943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059855    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:32.060057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.059866    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.060195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.060169    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.061557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.061534    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:32.062481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.062428    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:55:32.062579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.062518    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "kube-api-access-vp8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:32.063207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.063184    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo" pod "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" (UID: "01f1d84f-fcd1-4166-a7eb-11a5e6d63b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:55:32.098516 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.098430    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:32.161139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161112    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.161139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161136    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.161139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161146    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8ng\" (UniqueName: \"kubernetes.io/projected/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-kube-api-access-vp8ng\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.161375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161158    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.161375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161167    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.161375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161174    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.161375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.161182    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:32.432270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.432226    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"]
Oct 30 11:55:32.433201 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433177    5060 cpu_manager.go:414] "containerMap: removing container" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy"
Oct 30 11:55:32.433201 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433202    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy"
Oct 30 11:55:32.433379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433243    5060 cpu_manager.go:414] "containerMap: removing container" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy"
Oct 30 11:55:32.433379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433251    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy"
Oct 30 11:55:32.433379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433306    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c49ee66d-6660-4320-ab97-54c3165b0ea6" containerName="revoke-keys"
Oct 30 11:55:32.433379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433314    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49ee66d-6660-4320-ab97-54c3165b0ea6" containerName="revoke-keys"
Oct 30 11:55:32.433623 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433466    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy"
Oct 30 11:55:32.433623 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433487    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49ee66d-6660-4320-ab97-54c3165b0ea6" containerName="revoke-keys"
Oct 30 11:55:32.433623 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.433499    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerName="istio-proxy"
Oct 30 11:55:32.437962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.437926    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.456113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.456084    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-648839-openshift-default-dockercfg-qkdk6\""
Oct 30 11:55:32.458537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.458508    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"]
Oct 30 11:55:32.465122 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465087    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465344 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465320    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfvr\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465365    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465659 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465640    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465736 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465676    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465813 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465790    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465869 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465845    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.465930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.465914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.510054 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.509958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569538    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfvr\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569597    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569701    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569777    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569817    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569846    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.581483 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.569921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.593792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.583216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.593792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.583925    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.593792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.584416    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.593792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.585041    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.593792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.585271    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.621753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.612085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.625086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.625021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.626106 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.626038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.629211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.629139    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfvr\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr\") pod \"e2e-ipp-b-648839-openshift-default-66c78757c-sg68w\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") " pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.742855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.740841    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:32.758143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.757720    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:32.793742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.789193    5060 generic.go:356] "Generic (PLEG): container finished" podID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" containerID="094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a" exitCode=0
Oct 30 11:55:32.793742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.789323    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"
Oct 30 11:55:32.793742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.789343    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" event={"ID":"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76","Type":"ContainerDied","Data":"094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a"}
Oct 30 11:55:32.793742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.789658    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47" event={"ID":"01f1d84f-fcd1-4166-a7eb-11a5e6d63b76","Type":"ContainerDied","Data":"4cc1be2c077b32c1b41f6440370902b9b9c1e28b6edd7d54ddac17e36ec8499f"}
Oct 30 11:55:32.793742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.789709    5060 scope.go:122] "RemoveContainer" containerID="094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a"
Oct 30 11:55:32.814080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.813107    5060 scope.go:122] "RemoveContainer" containerID="9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"
Oct 30 11:55:32.828975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.828949    5060 scope.go:122] "RemoveContainer" containerID="094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a"
Oct 30 11:55:32.829364 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:32.829338    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a\": container with ID starting with 094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a not found: ID does not exist" containerID="094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a"
Oct 30 11:55:32.829528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.829372    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a"} err="failed to get container status \"094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a\": rpc error: code = NotFound desc = could not find container \"094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a\": container with ID starting with 094f8a9eed79f33d67b0d105ca3876387d33fcec1734f68798e3b7a1fa63078a not found: ID does not exist"
Oct 30 11:55:32.829528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.829398    5060 scope.go:122] "RemoveContainer" containerID="9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"
Oct 30 11:55:32.829788 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:32.829760    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155\": container with ID starting with 9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155 not found: ID does not exist" containerID="9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"
Oct 30 11:55:32.829867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.829798    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155"} err="failed to get container status \"9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155\": rpc error: code = NotFound desc = could not find container \"9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155\": container with ID starting with 9c53078eb457a99131867387420328341c27a85b31b774ea4f03e2b2bec4c155 not found: ID does not exist"
Oct 30 11:55:32.845976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.845867    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"]
Oct 30 11:55:32.851936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:32.851904    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84fed1-openshift-default-7f78d7f9d9-mtj47"]
Oct 30 11:55:33.135067 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:33.135037    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d4ce99_923c_4bdd_b10b_b8a3ab2601e2.slice/crio-63e6146e58608187005ae3f9978947128b6d3a2d801bc3303d61c7a74c3e06ae WatchSource:0}: Error finding container 63e6146e58608187005ae3f9978947128b6d3a2d801bc3303d61c7a74c3e06ae: Status 404 returned error can't find the container with id 63e6146e58608187005ae3f9978947128b6d3a2d801bc3303d61c7a74c3e06ae
Oct 30 11:55:33.136242 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.136220    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"]
Oct 30 11:55:33.138263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.138231    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:33.138351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.138295    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:33.138351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.138326    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:55:33.356020 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.355978    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:33.537478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.536419    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:33.915115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.914796    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" event={"ID":"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2","Type":"ContainerStarted","Data":"e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"}
Oct 30 11:55:33.915115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.914935    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" event={"ID":"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2","Type":"ContainerStarted","Data":"63e6146e58608187005ae3f9978947128b6d3a2d801bc3303d61c7a74c3e06ae"}
Oct 30 11:55:33.960629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:33.960569    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podStartSLOduration=1.960551464 podStartE2EDuration="1.960551464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:33.959788292 +0000 UTC m=+1537.754672786" watchObservedRunningTime="2026-10-30 11:55:33.960551464 +0000 UTC m=+1537.755435958"
Oct 30 11:55:34.335359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:34.335327    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f1d84f-fcd1-4166-a7eb-11a5e6d63b76" path="/var/lib/kubelet/pods/01f1d84f-fcd1-4166-a7eb-11a5e6d63b76/volumes"
Oct 30 11:55:34.626491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:34.626350    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:34.758296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:34.758261    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:34.765576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:34.765546    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:34.923041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:34.922943    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:34.924185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:34.924162    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:55:36.462368 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:36.462332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:36.642626 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:36.642587    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:36.954445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:36.945754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:38.019947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:38.019914    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:38.165465 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:38.165409    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:39.084898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.084859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:39.236976 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.236947    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:39.304898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.304864    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"]
Oct 30 11:55:39.310650 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.310623    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.329349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.329312    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"]
Oct 30 11:55:39.335337 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.335266    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-648839-dockercfg-9fdz4\""
Oct 30 11:55:39.335337 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.335267    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-648839\""
Oct 30 11:55:39.382425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.382382    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.382729 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.382700    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csws\" (UniqueName: \"kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws\") pod \"payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.400010 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.399924    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:39.402112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.402083    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"]
Oct 30 11:55:39.407882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.407855    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.455493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.452273    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"]
Oct 30 11:55:39.554476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.496495    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.554476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.496553    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csws\" (UniqueName: \"kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws\") pod \"payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.568517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.556896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csws\" (UniqueName: \"kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws\") pod \"payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.590537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.569609    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.601208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.601146    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwlm\" (UniqueName: \"kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm\") pod \"payload-processing-e2e-ipp-b-648839-74684687fd-4ks77\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.601746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.601722    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648839-74684687fd-4ks77\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.628525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.628312    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:39.703318 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.703284    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwlm\" (UniqueName: \"kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm\") pod \"payload-processing-e2e-ipp-b-648839-74684687fd-4ks77\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.703576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.703528    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648839-74684687fd-4ks77\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.704292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.704271    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648839-74684687fd-4ks77\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.730226 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.729649    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwlm\" (UniqueName: \"kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm\") pod \"payload-processing-e2e-ipp-b-648839-74684687fd-4ks77\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.928975 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.928880    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:39.948976 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:39.948941    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aba707_5e79_414b_a4e2_563b25d37ce8.slice/crio-79f0e8683c38197143518d0dfb2d7152d7fd267c0c663bce668199408a922efa WatchSource:0}: Error finding container 79f0e8683c38197143518d0dfb2d7152d7fd267c0c663bce668199408a922efa: Status 404 returned error can't find the container with id 79f0e8683c38197143518d0dfb2d7152d7fd267c0c663bce668199408a922efa
Oct 30 11:55:39.950389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.950365    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"]
Oct 30 11:55:39.972202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:39.972170    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl" event={"ID":"58aba707-5e79-414b-a4e2-563b25d37ce8","Type":"ContainerStarted","Data":"79f0e8683c38197143518d0dfb2d7152d7fd267c0c663bce668199408a922efa"}
Oct 30 11:55:40.242058 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:40.242023    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dbbdb7_9167_4286_a281_000bd9463600.slice/crio-d69818704b1fd0c0fe0cbdb559e363a4c94ecd19246cdcbaa1ca31e573e6ffb7 WatchSource:0}: Error finding container d69818704b1fd0c0fe0cbdb559e363a4c94ecd19246cdcbaa1ca31e573e6ffb7: Status 404 returned error can't find the container with id d69818704b1fd0c0fe0cbdb559e363a4c94ecd19246cdcbaa1ca31e573e6ffb7
Oct 30 11:55:40.243212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.243189    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"]
Oct 30 11:55:40.764241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.764213    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:55:40.764436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.764280    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:55:40.983166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.983130    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77" event={"ID":"10dbbdb7-9167-4286-a281-000bd9463600","Type":"ContainerStarted","Data":"a7ffe7ae5aa5e640278d088eac91e1020e432114c7e0f7476135741a0a812cfb"}
Oct 30 11:55:40.983380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.983191    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77" event={"ID":"10dbbdb7-9167-4286-a281-000bd9463600","Type":"ContainerStarted","Data":"d69818704b1fd0c0fe0cbdb559e363a4c94ecd19246cdcbaa1ca31e573e6ffb7"}
Oct 30 11:55:40.983380 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.983237    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:40.985217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.985186    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl" event={"ID":"58aba707-5e79-414b-a4e2-563b25d37ce8","Type":"ContainerStarted","Data":"53d4e855bfee717cf3cb84d0e49b0326d179fb228b23f6121eee0e8e4540c91f"}
Oct 30 11:55:40.985362 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:40.985250    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:41.003110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:41.003077    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:41.026146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:41.025036    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77" podStartSLOduration=2.025017169 podStartE2EDuration="2.025017169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:41.024243207 +0000 UTC m=+1544.819127712" watchObservedRunningTime="2026-10-30 11:55:41.025017169 +0000 UTC m=+1544.819901663"
Oct 30 11:55:41.070229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:41.070136    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl" podStartSLOduration=2.070119747 podStartE2EDuration="2.070119747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:41.067370008 +0000 UTC m=+1544.862254503" watchObservedRunningTime="2026-10-30 11:55:41.070119747 +0000 UTC m=+1544.865004241"
Oct 30 11:55:43.478762 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:43.478724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:45.740231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:45.740187    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:47.589032 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:47.588991    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:49.422063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:49.421973    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:49.698960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:49.698925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:49.860120 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:49.860072    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:55:50.316991 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.316943    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"]
Oct 30 11:55:50.328899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.328866    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"]
Oct 30 11:55:50.329096 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.328990    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.345279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.345231    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-caa7a0\"/\"kube-root-ca.crt\""
Oct 30 11:55:50.345279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.345255    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-caa7a0\"/\"default-dockercfg-wg6zg\""
Oct 30 11:55:50.345639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.345316    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-caa7a0\"/\"openshift-service-ca.crt\""
Oct 30 11:55:50.345639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.345360    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-caa7a0\"/\"ipp-route-caa7a0-kserve-self-signed-certs\""
Oct 30 11:55:50.381864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.381831    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:50.458018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.457982    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.458595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.458066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.458595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.458225    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.458595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.458318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jsn\" (UniqueName: \"kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.458595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.458349    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.458595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.458408    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.560221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.560406 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560287    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jsn\" (UniqueName: \"kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.560406 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560328    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.560406 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.560632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560482    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.560632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560560    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.561082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.560874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.561082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.561019    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.561082 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.561031    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.563423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.563397    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.563423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.563416    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.585894 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.585821    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jsn\" (UniqueName: \"kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn\") pod \"ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") " pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:50.660477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:50.652121    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:51.069320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:51.069215    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"]
Oct 30 11:55:51.073444 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:55:51.072881    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c717b3f_0861_454a_9789_a45c7aed058c.slice/crio-dcb2047c53e098f0032e99f61f712da5d2636a919401a69a5fba8ad457da9734 WatchSource:0}: Error finding container dcb2047c53e098f0032e99f61f712da5d2636a919401a69a5fba8ad457da9734: Status 404 returned error can't find the container with id dcb2047c53e098f0032e99f61f712da5d2636a919401a69a5fba8ad457da9734
Oct 30 11:55:51.579101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:51.579066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:51.730031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:51.729999    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:51.908572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:51.908471    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:52.000064 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.000031    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:55:52.000367 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.000348    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:55:52.101501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.100412    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn" event={"ID":"9c717b3f-0861-454a-9789-a45c7aed058c","Type":"ContainerStarted","Data":"4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3"}
Oct 30 11:55:52.101501 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.100526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn" event={"ID":"9c717b3f-0861-454a-9789-a45c7aed058c","Type":"ContainerStarted","Data":"dcb2047c53e098f0032e99f61f712da5d2636a919401a69a5fba8ad457da9734"}
Oct 30 11:55:52.164412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.164300    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn" podStartSLOduration=2.164285049 podStartE2EDuration="2.164285049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:52.161152852 +0000 UTC m=+1555.956037347" watchObservedRunningTime="2026-10-30 11:55:52.164285049 +0000 UTC m=+1555.959169542"
Oct 30 11:55:52.649982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.649940    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"]
Oct 30 11:55:52.652616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.652583    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:55:52.688624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.688577    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"]
Oct 30 11:55:52.721205 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.721170    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:52.892753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:52.892716    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:53.045758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:53.045724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:54.000120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:54.000085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:55.918200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:55.918152    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:57.007984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:57.007941    5060 scope.go:122] "RemoveContainer" containerID="833c12e20926fbd799e9eee4a77751d212f4cfb4191e55631ff6d09b125738ce"
Oct 30 11:55:57.650983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:57.650929    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj" podUID="1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" containerName="payload-processing" containerID="cri-o://8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49" gracePeriod=25
Oct 30 11:55:57.689124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:57.689072    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj" podUID="ab6c77d4-9990-48ea-82f7-7b65735b346e" containerName="payload-pre-processing" containerID="cri-o://1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8" gracePeriod=25
Oct 30 11:55:57.996798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:57.996766    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:55:58.045248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.045220    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:55:58.107950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.107909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq\" (UniqueName: \"kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq\") pod \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") "
Oct 30 11:55:58.108204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.108028    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume\") pod \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\" (UID: \"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4\") "
Oct 30 11:55:58.108968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.108938    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume" pod "1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" (UID: "1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:58.111574 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.111539    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq" pod "1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" (UID: "1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4"). InnerVolumeSpecName "kube-api-access-jqlgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:58.172815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.167277    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:55:58.191412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.189110    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" containerID="8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49" exitCode=0
Oct 30 11:55:58.191412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.189158    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"
Oct 30 11:55:58.191412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.189287    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj" event={"ID":"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4","Type":"ContainerDied","Data":"8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49"}
Oct 30 11:55:58.191412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.189329    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj" event={"ID":"1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4","Type":"ContainerDied","Data":"0e96f60d89a96a0c55195e46b9534db1ee1c0b6a93d95de7f95e735f826616b5"}
Oct 30 11:55:58.191412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.189354    5060 scope.go:122] "RemoveContainer" containerID="8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49"
Oct 30 11:55:58.195617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.195044    5060 generic.go:356] "Generic (PLEG): container finished" podID="ab6c77d4-9990-48ea-82f7-7b65735b346e" containerID="1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8" exitCode=0
Oct 30 11:55:58.195617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.195149    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj" event={"ID":"ab6c77d4-9990-48ea-82f7-7b65735b346e","Type":"ContainerDied","Data":"1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8"}
Oct 30 11:55:58.195617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.195251    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj" event={"ID":"ab6c77d4-9990-48ea-82f7-7b65735b346e","Type":"ContainerDied","Data":"bd576246ddaef1106330d55776fa929804b63e213c8bd802299b55b90957ae33"}
Oct 30 11:55:58.195617 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.195322    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"
Oct 30 11:55:58.210271 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.210029    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd\" (UniqueName: \"kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd\") pod \"ab6c77d4-9990-48ea-82f7-7b65735b346e\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") "
Oct 30 11:55:58.210511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.210355    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume\") pod \"ab6c77d4-9990-48ea-82f7-7b65735b346e\" (UID: \"ab6c77d4-9990-48ea-82f7-7b65735b346e\") "
Oct 30 11:55:58.211128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.211098    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqlgq\" (UniqueName: \"kubernetes.io/projected/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-kube-api-access-jqlgq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:58.211253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.211133    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:58.211554 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.211517    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume" pod "ab6c77d4-9990-48ea-82f7-7b65735b346e" (UID: "ab6c77d4-9990-48ea-82f7-7b65735b346e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:55:58.217792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.217755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd" pod "ab6c77d4-9990-48ea-82f7-7b65735b346e" (UID: "ab6c77d4-9990-48ea-82f7-7b65735b346e"). InnerVolumeSpecName "kube-api-access-ql8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:55:58.218734 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.218712    5060 scope.go:122] "RemoveContainer" containerID="8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49"
Oct 30 11:55:58.219125 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:58.219103    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49\": container with ID starting with 8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49 not found: ID does not exist" containerID="8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49"
Oct 30 11:55:58.219202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.219185    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49"} err="failed to get container status \"8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49\": rpc error: code = NotFound desc = could not find container \"8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49\": container with ID starting with 8d1f1b69abb232ca19cc9494f8fd0a166f569ffa74b70b5d501eee2fe8f8fe49 not found: ID does not exist"
Oct 30 11:55:58.219248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.219205    5060 scope.go:122] "RemoveContainer" containerID="1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8"
Oct 30 11:55:58.231863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.231840    5060 scope.go:122] "RemoveContainer" containerID="1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8"
Oct 30 11:55:58.232209 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:55:58.232188    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8\": container with ID starting with 1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8 not found: ID does not exist" containerID="1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8"
Oct 30 11:55:58.232282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.232217    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8"} err="failed to get container status \"1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8\": rpc error: code = NotFound desc = could not find container \"1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8\": container with ID starting with 1bf5907bb3424da18a9ef459537630a0ed0412a8f1f6ae0370a930657ffbbee8 not found: ID does not exist"
Oct 30 11:55:58.288054 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.288015    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"]
Oct 30 11:55:58.295207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.295173    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-769cd9f99b-zz5dj"]
Oct 30 11:55:58.312765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.312730    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6c77d4-9990-48ea-82f7-7b65735b346e-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:58.312765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.312762    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8wd\" (UniqueName: \"kubernetes.io/projected/ab6c77d4-9990-48ea-82f7-7b65735b346e-kube-api-access-ql8wd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:55:58.335878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.335843    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" path="/var/lib/kubelet/pods/1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4/volumes"
Oct 30 11:55:58.540994 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.540948    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"]
Oct 30 11:55:58.544814 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:55:58.544787    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-8696448cbc-rc7dj"]
Oct 30 11:56:00.017776 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:00.017731    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:00.337092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:00.337003    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6c77d4-9990-48ea-82f7-7b65735b346e" path="/var/lib/kubelet/pods/ab6c77d4-9990-48ea-82f7-7b65735b346e/volumes"
Oct 30 11:56:00.653062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:00.652968    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:56:00.666283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:00.666251    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:56:01.243577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:01.243544    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:56:02.461719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:02.461683    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:04.207008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:04.206972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:04.347742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:04.347694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:04.505730 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:04.505632    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:06.106716 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:06.106676    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:07.885916 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:07.885859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:09.037937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:09.037902    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"]
Oct 30 11:56:09.050537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:09.050496    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97842b20-pvc8j"]
Oct 30 11:56:09.714817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:09.714781    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:10.336020 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:10.335986    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a430c672-5d59-431d-baed-bf67c11ea117" path="/var/lib/kubelet/pods/a430c672-5d59-431d-baed-bf67c11ea117/volumes"
Oct 30 11:56:12.066166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:12.066130    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:14.095961 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:14.095914    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:14.758126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:14.758087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:14.863662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:14.863620    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"]
Oct 30 11:56:14.875503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:14.875465    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5389d2-8bm54"]
Oct 30 11:56:14.901263 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:14.901226    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:15.045385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:15.045301    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:15.468901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:15.468870    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:16.335599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.335566    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49ee66d-6660-4320-ab97-54c3165b0ea6" path="/var/lib/kubelet/pods/c49ee66d-6660-4320-ab97-54c3165b0ea6/volumes"
Oct 30 11:56:16.898955 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.898910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:16.965320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.965285    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"]
Oct 30 11:56:16.965572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.965531    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy" containerID="cri-o://a40d799a8bce1b156551a1d9f4252fc3a0470abb3e4aacac01983ff2fd495da7" gracePeriod=30
Oct 30 11:56:16.977959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.977900    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"]
Oct 30 11:56:16.978490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.978440    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy" containerID="cri-o://89e75f6fe163cb3505290c0a727e301bffd356db63ebfe5e72820486bbf28665" gracePeriod=30
Oct 30 11:56:16.983042 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.983010    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"]
Oct 30 11:56:16.983268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:16.983220    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy" containerID="cri-o://e48a28c0d746134ed09b8413666cad52530553c1f7abffcdc29ffe68b5937365" gracePeriod=30
Oct 30 11:56:17.000872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:17.000838    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"]
Oct 30 11:56:17.001100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:17.001076    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy" containerID="cri-o://224f9160ba3c067a0b2ffd5a69f1b74c8d651039900eb77b92b7ac755614b56a" gracePeriod=30
Oct 30 11:56:18.726485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:18.725221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:19.856998 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:56:19.856961    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:56:20.132092 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.131987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:20.302441 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.300542    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:20.835497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.835417    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"]
Oct 30 11:56:20.836326 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.836297    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ab6c77d4-9990-48ea-82f7-7b65735b346e" containerName="payload-pre-processing"
Oct 30 11:56:20.836326 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.836322    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6c77d4-9990-48ea-82f7-7b65735b346e" containerName="payload-pre-processing"
Oct 30 11:56:20.836602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.836341    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" containerName="payload-processing"
Oct 30 11:56:20.836602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.836348    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" containerName="payload-processing"
Oct 30 11:56:20.836602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.836538    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6c77d4-9990-48ea-82f7-7b65735b346e" containerName="payload-pre-processing"
Oct 30 11:56:20.836602 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.836562    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4c51be-cb35-4bbd-a0aa-0651a5aa06e4" containerName="payload-processing"
Oct 30 11:56:20.840678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.840655    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:20.862539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.861729    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"]
Oct 30 11:56:20.865851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.865826    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eaee54\""
Oct 30 11:56:20.866012 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.865916    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eaee54-dockercfg-nr2td\""
Oct 30 11:56:20.908254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.908217    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s658d\" (UniqueName: \"kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:20.908611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:20.908581    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:21.009982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.009946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:21.010168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.010038    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s658d\" (UniqueName: \"kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:21.010738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.010714    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:21.038964 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.038929    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s658d\" (UniqueName: \"kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d\") pod \"payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:21.164891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.164820    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:21.451649 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.451614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:21.484976 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:56:21.484942    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32033b38_f509_4a6c_a7e6_0e7b122376e2.slice/crio-561b229f6d15ec941aab09f3b9df367da4841a45a5be096ef3a220ec04e64ea9 WatchSource:0}: Error finding container 561b229f6d15ec941aab09f3b9df367da4841a45a5be096ef3a220ec04e64ea9: Status 404 returned error can't find the container with id 561b229f6d15ec941aab09f3b9df367da4841a45a5be096ef3a220ec04e64ea9
Oct 30 11:56:21.485849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.485682    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"]
Oct 30 11:56:21.487817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.487797    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:56:21.898362 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:21.898268    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:22.070497 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.060711    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:22.310799 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.307325    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:22.495007 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.494497    5060 generic.go:356] "Generic (PLEG): container finished" podID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerID="e48a28c0d746134ed09b8413666cad52530553c1f7abffcdc29ffe68b5937365" exitCode=0
Oct 30 11:56:22.495007 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.494628    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" event={"ID":"55f70fc3-88ee-4ec1-9597-49e6413a827b","Type":"ContainerDied","Data":"e48a28c0d746134ed09b8413666cad52530553c1f7abffcdc29ffe68b5937365"}
Oct 30 11:56:22.496638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.496509    5060 generic.go:356] "Generic (PLEG): container finished" podID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerID="89e75f6fe163cb3505290c0a727e301bffd356db63ebfe5e72820486bbf28665" exitCode=0
Oct 30 11:56:22.496915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.496842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" event={"ID":"7fad95e2-b9b8-40ba-84ac-cff5670a363c","Type":"ContainerDied","Data":"89e75f6fe163cb3505290c0a727e301bffd356db63ebfe5e72820486bbf28665"}
Oct 30 11:56:22.501822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.501756    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg" event={"ID":"32033b38-f509-4a6c-a7e6-0e7b122376e2","Type":"ContainerStarted","Data":"561b229f6d15ec941aab09f3b9df367da4841a45a5be096ef3a220ec04e64ea9"}
Oct 30 11:56:22.507234 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.506897    5060 generic.go:356] "Generic (PLEG): container finished" podID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerID="a40d799a8bce1b156551a1d9f4252fc3a0470abb3e4aacac01983ff2fd495da7" exitCode=0
Oct 30 11:56:22.507234 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.506984    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" event={"ID":"ab5ceff7-b397-46cd-a04f-4aeb01146dce","Type":"ContainerDied","Data":"a40d799a8bce1b156551a1d9f4252fc3a0470abb3e4aacac01983ff2fd495da7"}
Oct 30 11:56:22.525328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.525290    5060 generic.go:356] "Generic (PLEG): container finished" podID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerID="224f9160ba3c067a0b2ffd5a69f1b74c8d651039900eb77b92b7ac755614b56a" exitCode=0
Oct 30 11:56:22.526056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.526001    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" event={"ID":"4b748992-0b9d-41ce-9986-83f3ed5296f7","Type":"ContainerDied","Data":"224f9160ba3c067a0b2ffd5a69f1b74c8d651039900eb77b92b7ac755614b56a"}
Oct 30 11:56:22.564817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.564763    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:56:22.652134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.652100    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:56:22.657079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.657009    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:56:22.663640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.663556    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:56:22.677678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677556    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677704    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677815    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677845    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677913    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.677992 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.677964    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.678248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.678057    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs\") pod \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\" (UID: \"ab5ceff7-b397-46cd-a04f-4aeb01146dce\") "
Oct 30 11:56:22.678745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.678644    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.679929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.679821    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.679929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.679838    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:22.679929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.679903    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.680286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.680260    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.681495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.681316    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:56:22.681760 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.681719    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "kube-api-access-b6c2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.682081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.682046    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.683119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.683080    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token" pod "ab5ceff7-b397-46cd-a04f-4aeb01146dce" (UID: "ab5ceff7-b397-46cd-a04f-4aeb01146dce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.779756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.779668    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.779756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.779727    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.779978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.779767    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.779978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.779803    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.780121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780102    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.780169 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780160    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.780253 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780238    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.780322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780308    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780364    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780395    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780472    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780513 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780506    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.780636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780539    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.780636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780583    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780615    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.780781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780659    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780693    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.780781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780722    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.780781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.780982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780814    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.780982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780845    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.780982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780887    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.780982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.780957    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token\") pod \"4b748992-0b9d-41ce-9986-83f3ed5296f7\" (UID: \"4b748992-0b9d-41ce-9986-83f3ed5296f7\") "
Oct 30 11:56:22.781185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781009    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo\") pod \"55f70fc3-88ee-4ec1-9597-49e6413a827b\" (UID: \"55f70fc3-88ee-4ec1-9597-49e6413a827b\") "
Oct 30 11:56:22.781185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781050    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.781185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781095    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.781185 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781141    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token\") pod \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\" (UID: \"7fad95e2-b9b8-40ba-84ac-cff5670a363c\") "
Oct 30 11:56:22.781951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781921    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781954    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781970    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6c2p\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-kube-api-access-b6c2p\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.781987    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.782002    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.782014    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.782026    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.782037    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.782049    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5ceff7-b397-46cd-a04f-4aeb01146dce-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.782560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.782534    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:22.783048 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.783023    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.785836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.784446    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.785836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.784777    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.785836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.785098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.785836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.785240    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.785836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.785409    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.787596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.787570    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.788620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.788026    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.788620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.788026    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:22.788620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.788223    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.788620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.788318    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.788620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.788391    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.792504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.791709    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.792504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.791904    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.792504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.792081    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.792824 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.792731    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:22.792990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.792967    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.793180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.793154    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.793696 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.793626    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:56:22.794261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.794228    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.795252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.795226    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "kube-api-access-v5k8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.795958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.795934    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "kube-api-access-5ndlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.800140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.800105    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "kube-api-access-76sgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:22.800140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.800104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo" pod "4b748992-0b9d-41ce-9986-83f3ed5296f7" (UID: "4b748992-0b9d-41ce-9986-83f3ed5296f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:56:22.801767 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.801738    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo" pod "55f70fc3-88ee-4ec1-9597-49e6413a827b" (UID: "55f70fc3-88ee-4ec1-9597-49e6413a827b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:56:22.802532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.802508    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy" pod "7fad95e2-b9b8-40ba-84ac-cff5670a363c" (UID: "7fad95e2-b9b8-40ba-84ac-cff5670a363c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:56:22.883207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883168    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883202    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b748992-0b9d-41ce-9986-83f3ed5296f7-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883207 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883211    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883222    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883233    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883241    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883250    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883258    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5k8m\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-kube-api-access-v5k8m\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883267    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883275    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883283    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883290    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883297    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f70fc3-88ee-4ec1-9597-49e6413a827b-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883304    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b748992-0b9d-41ce-9986-83f3ed5296f7-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883313    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sgf\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-kube-api-access-76sgf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883320    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883328    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883335    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883342    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883351    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fad95e2-b9b8-40ba-84ac-cff5670a363c-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883358    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883366    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883374    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b748992-0b9d-41ce-9986-83f3ed5296f7-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883381    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f70fc3-88ee-4ec1-9597-49e6413a827b-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883389    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fad95e2-b9b8-40ba-84ac-cff5670a363c-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883396    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f70fc3-88ee-4ec1-9597-49e6413a827b-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:22.883468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:22.883404    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ndlm\" (UniqueName: \"kubernetes.io/projected/7fad95e2-b9b8-40ba-84ac-cff5670a363c-kube-api-access-5ndlm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:23.463932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.463898    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:23.539324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.539296    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"
Oct 30 11:56:23.539537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.539327    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb" event={"ID":"55f70fc3-88ee-4ec1-9597-49e6413a827b","Type":"ContainerDied","Data":"4667fb4d6f62c434565e2d9e8dc7c80b0e5fdc86be49c6c0a31f711524a776f6"}
Oct 30 11:56:23.539537 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.539410    5060 scope.go:122] "RemoveContainer" containerID="e48a28c0d746134ed09b8413666cad52530553c1f7abffcdc29ffe68b5937365"
Oct 30 11:56:23.541087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.541057    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr" event={"ID":"7fad95e2-b9b8-40ba-84ac-cff5670a363c","Type":"ContainerDied","Data":"9a253ef13d20aaa0628e1406d63e2fb8e189a3f18d7690ad3ab5fdcd1188d957"}
Oct 30 11:56:23.541194 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.541160    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"
Oct 30 11:56:23.545868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.545776    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt" event={"ID":"ab5ceff7-b397-46cd-a04f-4aeb01146dce","Type":"ContainerDied","Data":"56abb5a8cdf58b8abcd8961daeaac196cbfffdd847051127ccbe43ae781b9110"}
Oct 30 11:56:23.546007 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.545886    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"
Oct 30 11:56:23.553056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.552858    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g" event={"ID":"4b748992-0b9d-41ce-9986-83f3ed5296f7","Type":"ContainerDied","Data":"dcd460c6e64ca20b1e91d716b2d5e1b40ff8ecd1a80d393e92ead0e486dc0fb2"}
Oct 30 11:56:23.553056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.552967    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"
Oct 30 11:56:23.555911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.555891    5060 scope.go:122] "RemoveContainer" containerID="89e75f6fe163cb3505290c0a727e301bffd356db63ebfe5e72820486bbf28665"
Oct 30 11:56:23.570639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.570617    5060 scope.go:122] "RemoveContainer" containerID="a40d799a8bce1b156551a1d9f4252fc3a0470abb3e4aacac01983ff2fd495da7"
Oct 30 11:56:23.590743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.590722    5060 scope.go:122] "RemoveContainer" containerID="224f9160ba3c067a0b2ffd5a69f1b74c8d651039900eb77b92b7ac755614b56a"
Oct 30 11:56:23.652923 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.652887    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"]
Oct 30 11:56:23.658317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.658288    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-bc7pr"]
Oct 30 11:56:23.694791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.694755    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"]
Oct 30 11:56:23.697794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.697764    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z582g"]
Oct 30 11:56:23.732596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.732497    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"]
Oct 30 11:56:23.738594 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.738564    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-8f7zb"]
Oct 30 11:56:23.775218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.775185    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"]
Oct 30 11:56:23.784091 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:23.784059    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-7r2nt"]
Oct 30 11:56:24.086306 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.086215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:24.336512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.336428    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" path="/var/lib/kubelet/pods/4b748992-0b9d-41ce-9986-83f3ed5296f7/volumes"
Oct 30 11:56:24.336958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.336943    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" path="/var/lib/kubelet/pods/55f70fc3-88ee-4ec1-9597-49e6413a827b/volumes"
Oct 30 11:56:24.337340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.337328    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" path="/var/lib/kubelet/pods/7fad95e2-b9b8-40ba-84ac-cff5670a363c/volumes"
Oct 30 11:56:24.338059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.338045    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" path="/var/lib/kubelet/pods/ab5ceff7-b397-46cd-a04f-4aeb01146dce/volumes"
Oct 30 11:56:24.564503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.564435    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg" event={"ID":"32033b38-f509-4a6c-a7e6-0e7b122376e2","Type":"ContainerStarted","Data":"82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653"}
Oct 30 11:56:24.564951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.564664    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:24.609784 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:24.609657    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg" podStartSLOduration=2.574321923 podStartE2EDuration="4.609636208s" totalImagesPullingTime="2.035314285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:24.605867615 +0000 UTC m=+1588.400752110" watchObservedRunningTime="2026-10-30 11:56:24.609636208 +0000 UTC m=+1588.404520702"
Oct 30 11:56:25.281314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:25.281236    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:26.945874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:26.945843    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:27.841044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.841000    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"]
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842257    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842281    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842347    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842354    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842373    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842380    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842407    5060 cpu_manager.go:414] "containerMap: removing container" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842414    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842749    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ceff7-b397-46cd-a04f-4aeb01146dce" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842778    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fad95e2-b9b8-40ba-84ac-cff5670a363c" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842789    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b748992-0b9d-41ce-9986-83f3ed5296f7" containerName="istio-proxy"
Oct 30 11:56:27.845269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.842806    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f70fc3-88ee-4ec1-9597-49e6413a827b" containerName="istio-proxy"
Oct 30 11:56:27.853560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.851786    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"]
Oct 30 11:56:27.853560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.851909    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:27.868187 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.868160    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:56:27.904480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.883660    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:56:27.941740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.938931    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:56:27.941740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.939832    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:56:27.941740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.941082    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:27.941740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.941251    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:27.941740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:27.941393    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stntl\" (UniqueName: \"kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.052644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.051465    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.052644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.051540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.052644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.051654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stntl\" (UniqueName: \"kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.059282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.059244    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:28.059803 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.059781    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.060009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.059994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.084477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.082135    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stntl\" (UniqueName: \"kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl\") pod \"maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.172087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.172010    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:28.655153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.648936    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"]
Oct 30 11:56:28.668267 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:28.668090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg" event={"ID":"cd759ddf-324a-47be-bb76-0c2b175c27b2","Type":"ContainerStarted","Data":"f4ccbb430bf4479ee33bcbff0ef64445067615126fa41a20ef57cabdde9ab0bd"}
Oct 30 11:56:29.690029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:29.688953    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg" event={"ID":"cd759ddf-324a-47be-bb76-0c2b175c27b2","Type":"ContainerStarted","Data":"a10cdca73403c8abd1a0361f82ca7dc55fde7cf542ee622d0e96486186eeeb3d"}
Oct 30 11:56:29.740709 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:29.740644    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg" podStartSLOduration=2.740624109 podStartE2EDuration="2.740624109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:29.737697566 +0000 UTC m=+1593.532582061" watchObservedRunningTime="2026-10-30 11:56:29.740624109 +0000 UTC m=+1593.535508604"
Oct 30 11:56:29.924563 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:29.924377    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:30.717488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:30.717277    5060 generic.go:356] "Generic (PLEG): container finished" podID="cd759ddf-324a-47be-bb76-0c2b175c27b2" containerID="a10cdca73403c8abd1a0361f82ca7dc55fde7cf542ee622d0e96486186eeeb3d" exitCode=0
Oct 30 11:56:30.717488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:30.717435    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg" event={"ID":"cd759ddf-324a-47be-bb76-0c2b175c27b2","Type":"ContainerDied","Data":"a10cdca73403c8abd1a0361f82ca7dc55fde7cf542ee622d0e96486186eeeb3d"}
Oct 30 11:56:32.146932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.146901    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:32.163425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.163378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca\") pod \"cd759ddf-324a-47be-bb76-0c2b175c27b2\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") "
Oct 30 11:56:32.163608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.163470    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl\" (UniqueName: \"kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl\") pod \"cd759ddf-324a-47be-bb76-0c2b175c27b2\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") "
Oct 30 11:56:32.163608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.163571    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca\") pod \"cd759ddf-324a-47be-bb76-0c2b175c27b2\" (UID: \"cd759ddf-324a-47be-bb76-0c2b175c27b2\") "
Oct 30 11:56:32.164320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.164231    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca" pod "cd759ddf-324a-47be-bb76-0c2b175c27b2" (UID: "cd759ddf-324a-47be-bb76-0c2b175c27b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:32.165661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.165635    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca" pod "cd759ddf-324a-47be-bb76-0c2b175c27b2" (UID: "cd759ddf-324a-47be-bb76-0c2b175c27b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:56:32.165756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.165736    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl" pod "cd759ddf-324a-47be-bb76-0c2b175c27b2" (UID: "cd759ddf-324a-47be-bb76-0c2b175c27b2"). InnerVolumeSpecName "kube-api-access-stntl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:32.175274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.175247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:32.266476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.265507    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:32.266476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.265552    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stntl\" (UniqueName: \"kubernetes.io/projected/cd759ddf-324a-47be-bb76-0c2b175c27b2-kube-api-access-stntl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:32.266476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.265568    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd759ddf-324a-47be-bb76-0c2b175c27b2-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:32.771301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.771268    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"
Oct 30 11:56:32.771301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.771285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg" event={"ID":"cd759ddf-324a-47be-bb76-0c2b175c27b2","Type":"ContainerDied","Data":"f4ccbb430bf4479ee33bcbff0ef64445067615126fa41a20ef57cabdde9ab0bd"}
Oct 30 11:56:32.771622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:32.771361    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ccbb430bf4479ee33bcbff0ef64445067615126fa41a20ef57cabdde9ab0bd"
Oct 30 11:56:34.046640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.046592    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:34.497517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.494077    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"]
Oct 30 11:56:34.497517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.495003    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cd759ddf-324a-47be-bb76-0c2b175c27b2" containerName="revoke-keys"
Oct 30 11:56:34.497517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.495023    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd759ddf-324a-47be-bb76-0c2b175c27b2" containerName="revoke-keys"
Oct 30 11:56:34.497517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.495177    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd759ddf-324a-47be-bb76-0c2b175c27b2" containerName="revoke-keys"
Oct 30 11:56:34.499681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.499654    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.524865 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.524828    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"]
Oct 30 11:56:34.525327 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.525306    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3f5339-openshift-default-dockercfg-22rq8\""
Oct 30 11:56:34.586100 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.586050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:34.602312 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602266    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602337    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602377    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602424    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602571 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602520    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602601    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602634    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlkf\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.602795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.602702    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708360    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708401    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708447    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708529    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708649    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlkf\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708685    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.708885 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.708715    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.712478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.709671    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.712478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.709922    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.712478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.710135    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.712478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.711031    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.742112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.714082    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.742112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.719749    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.742112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.725920    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.767530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.744181    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:34.767530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.745590    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlkf\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.767530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.746003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token\") pod \"e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") " pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:34.843480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:34.842715    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:35.213836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.213787    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"]
Oct 30 11:56:35.218532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.218490    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:35.218641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.218590    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:35.218680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.218640    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:35.590531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.578764    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:56:35.842190 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.842092    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" event={"ID":"ba0f9611-6af9-4819-b351-d8cad17c5866","Type":"ContainerStarted","Data":"ebd2c0a3260fbb35afa0c7558f42f5485a9d9073f1e564a4ebce5f0cf7fd772e"}
Oct 30 11:56:35.842190 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.842171    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" event={"ID":"ba0f9611-6af9-4819-b351-d8cad17c5866","Type":"ContainerStarted","Data":"b1208add0412a0ee4928cf402b7c3aeec475835bf34773f3ae71d5cb76ee805d"}
Oct 30 11:56:35.844830 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.844799    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:35.891478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.887591    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:35.974027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.972627    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:56:35.974027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:35.972706    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:56:36.852162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:36.852127    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:36.911855 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:36.911790    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podStartSLOduration=2.91176885 podStartE2EDuration="2.91176885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:35.91542181 +0000 UTC m=+1599.710306305" watchObservedRunningTime="2026-10-30 11:56:36.91176885 +0000 UTC m=+1600.706653346"
Oct 30 11:56:37.694424 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.694383    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"]
Oct 30 11:56:37.733087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.733055    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"]
Oct 30 11:56:37.879528 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.878569    5060 generic.go:356] "Generic (PLEG): container finished" podID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerID="bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993" exitCode=0
Oct 30 11:56:37.882146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.880239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" event={"ID":"53c7d92f-8eef-4be9-b3d7-e0e877130074","Type":"ContainerDied","Data":"bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"}
Oct 30 11:56:37.882146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.880304    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:37.882146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.880762    5060 scope.go:122] "RemoveContainer" containerID="bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"
Oct 30 11:56:37.885393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.883292    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:56:37.885393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.884636    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:37.885393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.884738    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:37.885393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.884798    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:37.888648 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:37.888622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.039616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.039586    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.189560 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.189252    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.337862 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.337819    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.508476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.506921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.584476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.584170    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"]
Oct 30 11:56:38.604775 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.604749    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"]
Oct 30 11:56:38.759900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.759593    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.922477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.922221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:38.922958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.922521    5060 generic.go:356] "Generic (PLEG): container finished" podID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerID="e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd" exitCode=0
Oct 30 11:56:38.922958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.922588    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" event={"ID":"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2","Type":"ContainerDied","Data":"e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"}
Oct 30 11:56:38.923079 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.923002    5060 scope.go:122] "RemoveContainer" containerID="e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"
Oct 30 11:56:38.925750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.925711    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:38.925887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.925811    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:38.925887 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.925858    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:56:38.927658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:38.927600    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" event={"ID":"53c7d92f-8eef-4be9-b3d7-e0e877130074","Type":"ContainerStarted","Data":"87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052"}
Oct 30 11:56:39.698792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:39.698757    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:56:39.699202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:39.699178    5060 patch_prober.go:29] interesting pod/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 11:56:39.699400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:39.699375    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 11:56:39.943861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:39.943816    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" event={"ID":"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2","Type":"ContainerStarted","Data":"21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303"}
Oct 30 11:56:40.047564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.047174    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:40.217478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.212334    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:40.237509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.235898    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"]
Oct 30 11:56:40.241284 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.241258    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.250972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.250939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cd4\" (UniqueName: \"kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4\") pod \"payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.251114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.251029    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.256902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.256874    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3f5339\""
Oct 30 11:56:40.257112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.257071    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3f5339-dockercfg-9mm4r\""
Oct 30 11:56:40.257243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.257218    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"]
Oct 30 11:56:40.435639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.404233    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.435639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.404573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cd4\" (UniqueName: \"kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4\") pod \"payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.437120 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.436839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.518479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.517997    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"]
Oct 30 11:56:40.532769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.532728    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cd4\" (UniqueName: \"kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4\") pod \"payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.556679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.556649    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"]
Oct 30 11:56:40.556882 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.556784    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.557474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.557433    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:40.642865 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.642556    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flnb\" (UniqueName: \"kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb\") pod \"payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.642865 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.642680    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.719526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.702224    5060 patch_prober.go:29] interesting pod/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 11:56:40.719526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.702283    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 11:56:40.737177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.737132    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:56:40.751373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.751324    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flnb\" (UniqueName: \"kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb\") pod \"payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.751575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.751478    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.752668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.752635    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.758872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.758839    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:56:40.766005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.765971    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 11:56:40.766179 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.766026    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 11:56:40.805482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.802579    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flnb\" (UniqueName: \"kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb\") pod \"payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.852101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.852056    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"]
Oct 30 11:56:40.857593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.857565    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:40.874285 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.874255    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:56:40.874493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.874390    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:56:40.874581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.874495    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:56:40.874644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.874585    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:56:40.942250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.942209    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"]
Oct 30 11:56:40.946627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.946598    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:40.955292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.955003    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:40.955292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.955061    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4nd\" (UniqueName: \"kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:40.955292 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.955207    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:40.962365 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:40.961239    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"]
Oct 30 11:56:40.962365 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:56:40.961368    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625faff0_053f_455c_8f59_cf8f99b29c53.slice/crio-c1da520b32e4b76790761c6ad5bcaefb525af2a46794ef5790d8452c8bbdb9ba WatchSource:0}: Error finding container c1da520b32e4b76790761c6ad5bcaefb525af2a46794ef5790d8452c8bbdb9ba: Status 404 returned error can't find the container with id c1da520b32e4b76790761c6ad5bcaefb525af2a46794ef5790d8452c8bbdb9ba
Oct 30 11:56:41.057362 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.057013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.057714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.057691    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.057998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.057980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.058405 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.058383    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4nd\" (UniqueName: \"kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.060633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.060610    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.098480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.093580    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4nd\" (UniqueName: \"kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd\") pod \"maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.200509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.197486    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:41.438221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.438184    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"]
Oct 30 11:56:41.640894 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.640860    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"]
Oct 30 11:56:41.732595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.720254    5060 patch_prober.go:29] interesting pod/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 11:56:41.732595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.720307    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 11:56:41.758683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.758586    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 11:56:41.758683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.758634    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 11:56:41.964243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.964204    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26" event={"ID":"96c2a45a-4224-48c1-aa0d-1cf4573d75bc","Type":"ContainerStarted","Data":"a55b220ff360d2335139d2f49c79db360e2ee1f20f9fed002d465a6608c22919"}
Oct 30 11:56:41.966700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.966670    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59" event={"ID":"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096","Type":"ContainerStarted","Data":"851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603"}
Oct 30 11:56:41.966869 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.966721    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59" event={"ID":"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096","Type":"ContainerStarted","Data":"e61980f98a2121d15ab994968aaa593e20b9413985687ac1a4c8ee4ddb3c7ef6"}
Oct 30 11:56:41.967610 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.967579    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:41.969241 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.969217    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m" event={"ID":"625faff0-053f-455c-8f59-cf8f99b29c53","Type":"ContainerStarted","Data":"b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421"}
Oct 30 11:56:41.969354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.969263    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m" event={"ID":"625faff0-053f-455c-8f59-cf8f99b29c53","Type":"ContainerStarted","Data":"c1da520b32e4b76790761c6ad5bcaefb525af2a46794ef5790d8452c8bbdb9ba"}
Oct 30 11:56:41.969416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:41.969384    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:42.038986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.038859    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59" podStartSLOduration=2.038839916 podStartE2EDuration="2.038839916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:42.031885802 +0000 UTC m=+1605.826770297" watchObservedRunningTime="2026-10-30 11:56:42.038839916 +0000 UTC m=+1605.833724412"
Oct 30 11:56:42.074180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.074124    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m" podStartSLOduration=2.074108975 podStartE2EDuration="2.074108975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:42.072531171 +0000 UTC m=+1605.867415665" watchObservedRunningTime="2026-10-30 11:56:42.074108975 +0000 UTC m=+1605.868993509"
Oct 30 11:56:42.695596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.695534    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77" podUID="10dbbdb7-9167-4286-a281-000bd9463600" containerName="payload-processing" containerID="cri-o://a7ffe7ae5aa5e640278d088eac91e1020e432114c7e0f7476135741a0a812cfb" gracePeriod=25
Oct 30 11:56:42.696374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.696192    5060 patch_prober.go:29] interesting pod/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 11:56:42.696611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.696584    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 11:56:42.738944 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.738686    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl" podUID="58aba707-5e79-414b-a4e2-563b25d37ce8" containerName="payload-pre-processing" containerID="cri-o://53d4e855bfee717cf3cb84d0e49b0326d179fb228b23f6121eee0e8e4540c91f" gracePeriod=25
Oct 30 11:56:42.755389 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.755351    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:42.761490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.761407    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 11:56:42.761747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:42.761708    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 11:56:43.025550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.018553    5060 generic.go:356] "Generic (PLEG): container finished" podID="58aba707-5e79-414b-a4e2-563b25d37ce8" containerID="53d4e855bfee717cf3cb84d0e49b0326d179fb228b23f6121eee0e8e4540c91f" exitCode=0
Oct 30 11:56:43.025550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.018748    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl" event={"ID":"58aba707-5e79-414b-a4e2-563b25d37ce8","Type":"ContainerDied","Data":"53d4e855bfee717cf3cb84d0e49b0326d179fb228b23f6121eee0e8e4540c91f"}
Oct 30 11:56:43.063754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.060268    5060 generic.go:356] "Generic (PLEG): container finished" podID="10dbbdb7-9167-4286-a281-000bd9463600" containerID="a7ffe7ae5aa5e640278d088eac91e1020e432114c7e0f7476135741a0a812cfb" exitCode=0
Oct 30 11:56:43.063754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.060408    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77" event={"ID":"10dbbdb7-9167-4286-a281-000bd9463600","Type":"ContainerDied","Data":"a7ffe7ae5aa5e640278d088eac91e1020e432114c7e0f7476135741a0a812cfb"}
Oct 30 11:56:43.074655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.070679    5060 generic.go:356] "Generic (PLEG): container finished" podID="96c2a45a-4224-48c1-aa0d-1cf4573d75bc" containerID="bc7cd897b49f3a84f5aa1fff93ce8627d3ec6c3a01b2a3f067d183e78763cd49" exitCode=0
Oct 30 11:56:43.074655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.070926    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26" event={"ID":"96c2a45a-4224-48c1-aa0d-1cf4573d75bc","Type":"ContainerDied","Data":"bc7cd897b49f3a84f5aa1fff93ce8627d3ec6c3a01b2a3f067d183e78763cd49"}
Oct 30 11:56:43.357197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.357127    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:56:43.445065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.444704    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:56:43.472132 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.472100    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm\" (UniqueName: \"kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm\") pod \"10dbbdb7-9167-4286-a281-000bd9463600\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") "
Oct 30 11:56:43.472323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.472313    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume\") pod \"10dbbdb7-9167-4286-a281-000bd9463600\" (UID: \"10dbbdb7-9167-4286-a281-000bd9463600\") "
Oct 30 11:56:43.472791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.472763    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume" pod "10dbbdb7-9167-4286-a281-000bd9463600" (UID: "10dbbdb7-9167-4286-a281-000bd9463600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:43.472939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.472907    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dbbdb7-9167-4286-a281-000bd9463600-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:43.474585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.474551    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm" pod "10dbbdb7-9167-4286-a281-000bd9463600" (UID: "10dbbdb7-9167-4286-a281-000bd9463600"). InnerVolumeSpecName "kube-api-access-vvwlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:43.573599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.573562    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws\" (UniqueName: \"kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws\") pod \"58aba707-5e79-414b-a4e2-563b25d37ce8\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") "
Oct 30 11:56:43.573832 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.573707    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume\") pod \"58aba707-5e79-414b-a4e2-563b25d37ce8\" (UID: \"58aba707-5e79-414b-a4e2-563b25d37ce8\") "
Oct 30 11:56:43.574229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.574144    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume" pod "58aba707-5e79-414b-a4e2-563b25d37ce8" (UID: "58aba707-5e79-414b-a4e2-563b25d37ce8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:43.574371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.574289    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwlm\" (UniqueName: \"kubernetes.io/projected/10dbbdb7-9167-4286-a281-000bd9463600-kube-api-access-vvwlm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:43.574371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.574303    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aba707-5e79-414b-a4e2-563b25d37ce8-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:43.576005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.575984    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws" pod "58aba707-5e79-414b-a4e2-563b25d37ce8" (UID: "58aba707-5e79-414b-a4e2-563b25d37ce8"). InnerVolumeSpecName "kube-api-access-2csws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:43.589499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.588531    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6" podUID="ca9fdf9d-d305-4e98-988f-8fd4badf33ea" containerName="payload-processing" containerID="cri-o://bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60" gracePeriod=25
Oct 30 11:56:43.620616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.616233    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td" podUID="4ca2f64c-870b-4bba-936f-ae9c76a3b0de" containerName="payload-pre-processing" containerID="cri-o://d2230380644f045a2cb831f428f2f2141d229351e4cea9db3bbac6b1d997a68b" gracePeriod=25
Oct 30 11:56:43.675751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.675708    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2csws\" (UniqueName: \"kubernetes.io/projected/58aba707-5e79-414b-a4e2-563b25d37ce8-kube-api-access-2csws\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:43.716503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.713281    5060 patch_prober.go:29] interesting pod/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 11:56:43.716503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.713348    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 11:56:43.759793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.759208    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 11:56:43.759793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:43.759279    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 11:56:44.149555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.149524    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:56:44.158350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.158309    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77" event={"ID":"10dbbdb7-9167-4286-a281-000bd9463600","Type":"ContainerDied","Data":"d69818704b1fd0c0fe0cbdb559e363a4c94ecd19246cdcbaa1ca31e573e6ffb7"}
Oct 30 11:56:44.158578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.158390    5060 scope.go:122] "RemoveContainer" containerID="a7ffe7ae5aa5e640278d088eac91e1020e432114c7e0f7476135741a0a812cfb"
Oct 30 11:56:44.158644 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.158599    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"
Oct 30 11:56:44.167533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.166995    5060 generic.go:356] "Generic (PLEG): container finished" podID="ca9fdf9d-d305-4e98-988f-8fd4badf33ea" containerID="bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60" exitCode=0
Oct 30 11:56:44.167533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.167209    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6" event={"ID":"ca9fdf9d-d305-4e98-988f-8fd4badf33ea","Type":"ContainerDied","Data":"bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60"}
Oct 30 11:56:44.167533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.167260    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6" event={"ID":"ca9fdf9d-d305-4e98-988f-8fd4badf33ea","Type":"ContainerDied","Data":"2fe59b0de8c83a14a9f7327e34883d4eb21f3a7ceeff46ad3e4e4b9a05ada171"}
Oct 30 11:56:44.167533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.167343    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"
Oct 30 11:56:44.173705 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.170063    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl" event={"ID":"58aba707-5e79-414b-a4e2-563b25d37ce8","Type":"ContainerDied","Data":"79f0e8683c38197143518d0dfb2d7152d7fd267c0c663bce668199408a922efa"}
Oct 30 11:56:44.173705 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.170164    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"
Oct 30 11:56:44.253765 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.253730    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume\") pod \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") "
Oct 30 11:56:44.254238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.254219    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5\" (UniqueName: \"kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5\") pod \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\" (UID: \"ca9fdf9d-d305-4e98-988f-8fd4badf33ea\") "
Oct 30 11:56:44.275485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.261634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5" pod "ca9fdf9d-d305-4e98-988f-8fd4badf33ea" (UID: "ca9fdf9d-d305-4e98-988f-8fd4badf33ea"). InnerVolumeSpecName "kube-api-access-vwsq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:44.275485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.262120    5060 scope.go:122] "RemoveContainer" containerID="bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60"
Oct 30 11:56:44.302529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.302491    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume" pod "ca9fdf9d-d305-4e98-988f-8fd4badf33ea" (UID: "ca9fdf9d-d305-4e98-988f-8fd4badf33ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:44.340549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.340496    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"]
Oct 30 11:56:44.349645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.348616    5060 generic.go:356] "Generic (PLEG): container finished" podID="4ca2f64c-870b-4bba-936f-ae9c76a3b0de" containerID="d2230380644f045a2cb831f428f2f2141d229351e4cea9db3bbac6b1d997a68b" exitCode=0
Oct 30 11:56:44.357664 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.357636    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsq5\" (UniqueName: \"kubernetes.io/projected/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-kube-api-access-vwsq5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:44.357807 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.357663    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9fdf9d-d305-4e98-988f-8fd4badf33ea-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:44.362398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.362243    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-74684687fd-4ks77"]
Oct 30 11:56:44.362398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.362285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td" event={"ID":"4ca2f64c-870b-4bba-936f-ae9c76a3b0de","Type":"ContainerDied","Data":"d2230380644f045a2cb831f428f2f2141d229351e4cea9db3bbac6b1d997a68b"}
Oct 30 11:56:44.373223 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.372953    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:56:44.412478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.408708    5060 scope.go:122] "RemoveContainer" containerID="bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60"
Oct 30 11:56:44.438760 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:56:44.438704    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60\": container with ID starting with bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60 not found: ID does not exist" containerID="bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60"
Oct 30 11:56:44.438970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.438756    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60"} err="failed to get container status \"bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60\": rpc error: code = NotFound desc = could not find container \"bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60\": container with ID starting with bc80c8b6bfa4db5df84346b328b4f495961715c0a3b5784b6870e56ff38eac60 not found: ID does not exist"
Oct 30 11:56:44.438970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.438788    5060 scope.go:122] "RemoveContainer" containerID="53d4e855bfee717cf3cb84d0e49b0326d179fb228b23f6121eee0e8e4540c91f"
Oct 30 11:56:44.441051 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.440987    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"]
Oct 30 11:56:44.445786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.445760    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-d96cf595f-qcrtl"]
Oct 30 11:56:44.521984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.521912    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"]
Oct 30 11:56:44.527890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.527798    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8bb87-57c496478-pg4g6"]
Oct 30 11:56:44.559643 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.559559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume\") pod \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") "
Oct 30 11:56:44.560260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.560020    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume" pod "4ca2f64c-870b-4bba-936f-ae9c76a3b0de" (UID: "4ca2f64c-870b-4bba-936f-ae9c76a3b0de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:44.560569 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.560513    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q\" (UniqueName: \"kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q\") pod \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\" (UID: \"4ca2f64c-870b-4bba-936f-ae9c76a3b0de\") "
Oct 30 11:56:44.561195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.561171    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:44.563224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.563182    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q" pod "4ca2f64c-870b-4bba-936f-ae9c76a3b0de" (UID: "4ca2f64c-870b-4bba-936f-ae9c76a3b0de"). InnerVolumeSpecName "kube-api-access-5zz2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:44.663496 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.662240    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz2q\" (UniqueName: \"kubernetes.io/projected/4ca2f64c-870b-4bba-936f-ae9c76a3b0de-kube-api-access-5zz2q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:44.716990 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.716942    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:44.759998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.759898    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 11:56:44.759998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.759958    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 11:56:44.762711 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.762681    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca\") pod \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") "
Oct 30 11:56:44.762846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.762778    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd\" (UniqueName: \"kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd\") pod \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") "
Oct 30 11:56:44.763031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.763013    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca\") pod \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\" (UID: \"96c2a45a-4224-48c1-aa0d-1cf4573d75bc\") "
Oct 30 11:56:44.763527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.763502    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca" pod "96c2a45a-4224-48c1-aa0d-1cf4573d75bc" (UID: "96c2a45a-4224-48c1-aa0d-1cf4573d75bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:56:44.765414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.765391    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca" pod "96c2a45a-4224-48c1-aa0d-1cf4573d75bc" (UID: "96c2a45a-4224-48c1-aa0d-1cf4573d75bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:56:44.765890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.765867    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd" pod "96c2a45a-4224-48c1-aa0d-1cf4573d75bc" (UID: "96c2a45a-4224-48c1-aa0d-1cf4573d75bc"). InnerVolumeSpecName "kube-api-access-gt4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:56:44.859652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.859563    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:56:44.861850 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.861804    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:56:44.865757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.865731    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:44.865757 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.865756    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4nd\" (UniqueName: \"kubernetes.io/projected/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-kube-api-access-gt4nd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:44.865983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:44.865765    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c2a45a-4224-48c1-aa0d-1cf4573d75bc-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:56:45.196519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.196480    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:45.360825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.360794    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"
Oct 30 11:56:45.360825 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.360818    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td" event={"ID":"4ca2f64c-870b-4bba-936f-ae9c76a3b0de","Type":"ContainerDied","Data":"256fc9487303bfe70762f03c67a734a92e442a74cf31800c5fa8c673a581a487"}
Oct 30 11:56:45.361094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.360868    5060 scope.go:122] "RemoveContainer" containerID="d2230380644f045a2cb831f428f2f2141d229351e4cea9db3bbac6b1d997a68b"
Oct 30 11:56:45.364033 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.364010    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"
Oct 30 11:56:45.364188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.364053    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26" event={"ID":"96c2a45a-4224-48c1-aa0d-1cf4573d75bc","Type":"ContainerDied","Data":"a55b220ff360d2335139d2f49c79db360e2ee1f20f9fed002d465a6608c22919"}
Oct 30 11:56:45.364188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.364111    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a55b220ff360d2335139d2f49c79db360e2ee1f20f9fed002d465a6608c22919"
Oct 30 11:56:45.436140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.436109    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"]
Oct 30 11:56:45.439484 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.439444    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8bb87-f46fd5847-lv5td"]
Oct 30 11:56:45.762800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.762770    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:56:45.763166 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.763143    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:56:45.763874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:45.763856    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:56:46.335774 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:46.335737    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10dbbdb7-9167-4286-a281-000bd9463600" path="/var/lib/kubelet/pods/10dbbdb7-9167-4286-a281-000bd9463600/volumes"
Oct 30 11:56:46.336200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:46.336183    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca2f64c-870b-4bba-936f-ae9c76a3b0de" path="/var/lib/kubelet/pods/4ca2f64c-870b-4bba-936f-ae9c76a3b0de/volumes"
Oct 30 11:56:46.336595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:46.336580    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aba707-5e79-414b-a4e2-563b25d37ce8" path="/var/lib/kubelet/pods/58aba707-5e79-414b-a4e2-563b25d37ce8/volumes"
Oct 30 11:56:46.336965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:46.336951    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9fdf9d-d305-4e98-988f-8fd4badf33ea" path="/var/lib/kubelet/pods/ca9fdf9d-d305-4e98-988f-8fd4badf33ea/volumes"
Oct 30 11:56:46.991323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:46.991287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:48.733530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:48.732942    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:49.838074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:49.838039    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:49.857744 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:56:49.857705    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:56:50.972186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:50.972143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:52.759655 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:52.759616    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:53.075437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:53.075341    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:56:54.300826 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:54.300783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:54.353143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:54.353109    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:56:54.482409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:54.482374    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:54.628400 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:54.628304    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:55.045083 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:55.045037    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:55.170304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:55.170260    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"]
Oct 30 11:56:55.181905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:55.181868    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4844f7-m5s26"]
Oct 30 11:56:55.211261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:55.211225    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:55.363788 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:55.363686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:56.336348 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:56.336311    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c2a45a-4224-48c1-aa0d-1cf4573d75bc" path="/var/lib/kubelet/pods/96c2a45a-4224-48c1-aa0d-1cf4573d75bc/volumes"
Oct 30 11:56:57.071584 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.071550    5060 scope.go:122] "RemoveContainer" containerID="550114098a6efb598c9c300d6b9155b180ab2f624e7ac82b26f9947f0d1c2322"
Oct 30 11:56:57.082731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.082707    5060 scope.go:122] "RemoveContainer" containerID="18a74094e739c139c5e5832852c66c771418ca4bc8a24a4a2a4d64936f22dab6"
Oct 30 11:56:57.093130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.093102    5060 scope.go:122] "RemoveContainer" containerID="172427c6cabcc9157665ad7a8becb5276ca50649c488ab7d76bd8ccbe01c2f70"
Oct 30 11:56:57.103408 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.103382    5060 scope.go:122] "RemoveContainer" containerID="e77e7337d8fe65b640f0a8ffaa3360442b0d23b2339f232dea7854b4bfdebd21"
Oct 30 11:56:57.113535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.113512    5060 scope.go:122] "RemoveContainer" containerID="c1a69edcdc61bdb23dd739109218839d5827565708a83ae88d096caa1a7a801a"
Oct 30 11:56:57.124022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.123998    5060 scope.go:122] "RemoveContainer" containerID="b0c3774e0ca63bc0801c82b62c2df86ead568893ccf810235ea7e580034a32e4"
Oct 30 11:56:57.458679 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:57.458641    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:56:59.774294 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:56:59.774257    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:01.557235 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:01.557202    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:02.901467 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:02.901408    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:03.063126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.063079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:03.296477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.296400    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"]
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297366    5060 cpu_manager.go:414] "containerMap: removing container" podUID="96c2a45a-4224-48c1-aa0d-1cf4573d75bc" containerName="revoke-keys"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297394    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c2a45a-4224-48c1-aa0d-1cf4573d75bc" containerName="revoke-keys"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297418    5060 cpu_manager.go:414] "containerMap: removing container" podUID="58aba707-5e79-414b-a4e2-563b25d37ce8" containerName="payload-pre-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297430    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aba707-5e79-414b-a4e2-563b25d37ce8" containerName="payload-pre-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297442    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ca9fdf9d-d305-4e98-988f-8fd4badf33ea" containerName="payload-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297469    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9fdf9d-d305-4e98-988f-8fd4badf33ea" containerName="payload-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297509    5060 cpu_manager.go:414] "containerMap: removing container" podUID="10dbbdb7-9167-4286-a281-000bd9463600" containerName="payload-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297519    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dbbdb7-9167-4286-a281-000bd9463600" containerName="payload-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297551    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4ca2f64c-870b-4bba-936f-ae9c76a3b0de" containerName="payload-pre-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297558    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca2f64c-870b-4bba-936f-ae9c76a3b0de" containerName="payload-pre-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297707    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca2f64c-870b-4bba-936f-ae9c76a3b0de" containerName="payload-pre-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297722    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c2a45a-4224-48c1-aa0d-1cf4573d75bc" containerName="revoke-keys"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297735    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aba707-5e79-414b-a4e2-563b25d37ce8" containerName="payload-pre-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297745    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9fdf9d-d305-4e98-988f-8fd4badf33ea" containerName="payload-processing"
Oct 30 11:57:03.303524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.297758    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dbbdb7-9167-4286-a281-000bd9463600" containerName="payload-processing"
Oct 30 11:57:03.324823 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.324788    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.342188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.342158    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-648839\""
Oct 30 11:57:03.367942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.367907    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-648839-dockercfg-f5tdv\""
Oct 30 11:57:03.369557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.369405    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.369872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.369801    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"]
Oct 30 11:57:03.370231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.370209    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4v59\" (UniqueName: \"kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59\") pod \"payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.472177 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.472129    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4v59\" (UniqueName: \"kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59\") pod \"payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.472651 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.472623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.474017 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.473989    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.496674 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.496644    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4v59\" (UniqueName: \"kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59\") pod \"payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:03.695014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:03.694920    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:04.006050 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.006014    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"]
Oct 30 11:57:04.032248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.032215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:04.175279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.175201    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:04.533922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.533888    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4" event={"ID":"94aefcd2-20db-4f74-8287-6e40074b270f","Type":"ContainerStarted","Data":"31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26"}
Oct 30 11:57:04.534112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.533935    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4" event={"ID":"94aefcd2-20db-4f74-8287-6e40074b270f","Type":"ContainerStarted","Data":"66423e80c4d5e90e68011df165a5ab5fc697c31884adb1f6481136cf0c14eb8c"}
Oct 30 11:57:04.534112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.534023    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:04.577296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:04.577231    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4" podStartSLOduration=1.577213438 podStartE2EDuration="1.577213438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:04.576142945 +0000 UTC m=+1628.371027439" watchObservedRunningTime="2026-10-30 11:57:04.577213438 +0000 UTC m=+1628.372097935"
Oct 30 11:57:05.708901 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:05.708867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:05.853262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:05.853221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:07.696037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:07.695973    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:10.201997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:10.201963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:12.006371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.006340    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:12.156905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.156868    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:12.357110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.357008    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:12.791626 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.791592    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:12.945332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.945296    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"]
Oct 30 11:57:12.950431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.950413    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:12.967794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.967765    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:57:12.967960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.967821    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:57:12.971053 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:12.971024    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"]
Oct 30 11:57:13.011947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.011906    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:57:13.012410 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.011962    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:57:13.033347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.033305    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.033564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.033468    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.033677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.033660    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5kfb\" (UniqueName: \"kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.135160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.135068    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.135350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.135213    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5kfb\" (UniqueName: \"kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.135350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.135253    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.136069 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.135989    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.137743 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.137721    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.164738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.164706    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5kfb\" (UniqueName: \"kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.264326 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.264285    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:13.575202 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:57:13.575172    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bba782_5bbc_469a_b020_afbb4e485c93.slice/crio-bcc7ce051fec519ba69e42c5a7a52c3bb4cf0fad66d331f7b84bd63f1742fc0c WatchSource:0}: Error finding container bcc7ce051fec519ba69e42c5a7a52c3bb4cf0fad66d331f7b84bd63f1742fc0c: Status 404 returned error can't find the container with id bcc7ce051fec519ba69e42c5a7a52c3bb4cf0fad66d331f7b84bd63f1742fc0c
Oct 30 11:57:13.577800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.577769    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"]
Oct 30 11:57:13.604066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:13.604031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh" event={"ID":"f9bba782-5bbc-469a-b020-afbb4e485c93","Type":"ContainerStarted","Data":"bcc7ce051fec519ba69e42c5a7a52c3bb4cf0fad66d331f7b84bd63f1742fc0c"}
Oct 30 11:57:14.614298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:14.614260    5060 generic.go:356] "Generic (PLEG): container finished" podID="f9bba782-5bbc-469a-b020-afbb4e485c93" containerID="f676cdd4fd736450835e0d0f8afa0ac691a66502b10dbd0c372e10f3a8c7317d" exitCode=0
Oct 30 11:57:14.614738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:14.614336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh" event={"ID":"f9bba782-5bbc-469a-b020-afbb4e485c93","Type":"ContainerDied","Data":"f676cdd4fd736450835e0d0f8afa0ac691a66502b10dbd0c372e10f3a8c7317d"}
Oct 30 11:57:14.649073 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:14.649034    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:15.543656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.543629    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:57:15.795897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.795830    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:15.876136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.876101    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca\") pod \"f9bba782-5bbc-469a-b020-afbb4e485c93\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") "
Oct 30 11:57:15.876352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.876217    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca\") pod \"f9bba782-5bbc-469a-b020-afbb4e485c93\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") "
Oct 30 11:57:15.876473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.876433    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb\" (UniqueName: \"kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb\") pod \"f9bba782-5bbc-469a-b020-afbb4e485c93\" (UID: \"f9bba782-5bbc-469a-b020-afbb4e485c93\") "
Oct 30 11:57:15.876728 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.876694    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca" pod "f9bba782-5bbc-469a-b020-afbb4e485c93" (UID: "f9bba782-5bbc-469a-b020-afbb4e485c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:15.877393 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.877371    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:15.878935 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.878907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca" pod "f9bba782-5bbc-469a-b020-afbb4e485c93" (UID: "f9bba782-5bbc-469a-b020-afbb4e485c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:57:15.879318 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.879299    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb" pod "f9bba782-5bbc-469a-b020-afbb4e485c93" (UID: "f9bba782-5bbc-469a-b020-afbb4e485c93"). InnerVolumeSpecName "kube-api-access-z5kfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:15.979115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.979071    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5kfb\" (UniqueName: \"kubernetes.io/projected/f9bba782-5bbc-469a-b020-afbb4e485c93-kube-api-access-z5kfb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:15.979115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:15.979117    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bba782-5bbc-469a-b020-afbb4e485c93-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:16.404052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:16.404011    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:16.633546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:16.633444    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh" event={"ID":"f9bba782-5bbc-469a-b020-afbb4e485c93","Type":"ContainerDied","Data":"bcc7ce051fec519ba69e42c5a7a52c3bb4cf0fad66d331f7b84bd63f1742fc0c"}
Oct 30 11:57:16.633546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:16.633481    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"
Oct 30 11:57:16.633546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:16.633510    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc7ce051fec519ba69e42c5a7a52c3bb4cf0fad66d331f7b84bd63f1742fc0c"
Oct 30 11:57:18.716621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:18.716578    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:19.858131 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:57:19.858092    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:57:20.411937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:20.411902    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:21.090766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.090726    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"]
Oct 30 11:57:21.091698 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.091674    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f9bba782-5bbc-469a-b020-afbb4e485c93" containerName="revoke-keys"
Oct 30 11:57:21.091815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.091699    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bba782-5bbc-469a-b020-afbb4e485c93" containerName="revoke-keys"
Oct 30 11:57:21.091912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.091899    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bba782-5bbc-469a-b020-afbb4e485c93" containerName="revoke-keys"
Oct 30 11:57:21.096022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.095995    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.110562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.110532    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"]
Oct 30 11:57:21.123500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.118356    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:57:21.123500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.118439    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:57:21.123500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.118600    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:57:21.123500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.123275    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:57:21.149665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.149631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6c6b\" (UniqueName: \"kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.149851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.149783    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.149851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.149835    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.251444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.251404    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6c6b\" (UniqueName: \"kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.251675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.251542    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.251675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.251579    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.252195 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.252170    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.254140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.254118    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.275932 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.275896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6c6b\" (UniqueName: \"kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b\") pod \"maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.413321 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.413223    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:21.723983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:21.723950    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"]
Oct 30 11:57:21.724320 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:57:21.724289    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4e53b4_e8b6_4d43_96e4_877eea255369.slice/crio-9eceef69989ffdf621698b4b1fa3220ecd2530a655eef9e45d38b3f8e0ce557b WatchSource:0}: Error finding container 9eceef69989ffdf621698b4b1fa3220ecd2530a655eef9e45d38b3f8e0ce557b: Status 404 returned error can't find the container with id 9eceef69989ffdf621698b4b1fa3220ecd2530a655eef9e45d38b3f8e0ce557b
Oct 30 11:57:22.329277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:22.329175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:22.696135 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:22.696094    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f4e53b4-e8b6-4d43-96e4-877eea255369" containerID="f05574f8b0bfc5fc24ec77c41ea5fb39beb360efd7b17dcaea24484a6d7071d5" exitCode=0
Oct 30 11:57:22.696323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:22.696161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs" event={"ID":"1f4e53b4-e8b6-4d43-96e4-877eea255369","Type":"ContainerDied","Data":"f05574f8b0bfc5fc24ec77c41ea5fb39beb360efd7b17dcaea24484a6d7071d5"}
Oct 30 11:57:22.696323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:22.696220    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs" event={"ID":"1f4e53b4-e8b6-4d43-96e4-877eea255369","Type":"ContainerStarted","Data":"9eceef69989ffdf621698b4b1fa3220ecd2530a655eef9e45d38b3f8e0ce557b"}
Oct 30 11:57:23.891268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.891234    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:23.985745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.985646    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca\") pod \"1f4e53b4-e8b6-4d43-96e4-877eea255369\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") "
Oct 30 11:57:23.986147 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.985746    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca\") pod \"1f4e53b4-e8b6-4d43-96e4-877eea255369\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") "
Oct 30 11:57:23.986147 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.985826    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b\" (UniqueName: \"kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b\") pod \"1f4e53b4-e8b6-4d43-96e4-877eea255369\" (UID: \"1f4e53b4-e8b6-4d43-96e4-877eea255369\") "
Oct 30 11:57:23.986334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.986285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca" pod "1f4e53b4-e8b6-4d43-96e4-877eea255369" (UID: "1f4e53b4-e8b6-4d43-96e4-877eea255369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:23.987279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.987237    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:23.988530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.988508    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b" pod "1f4e53b4-e8b6-4d43-96e4-877eea255369" (UID: "1f4e53b4-e8b6-4d43-96e4-877eea255369"). InnerVolumeSpecName "kube-api-access-b6c6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:23.988778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:23.988761    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca" pod "1f4e53b4-e8b6-4d43-96e4-877eea255369" (UID: "1f4e53b4-e8b6-4d43-96e4-877eea255369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:57:24.088962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:24.088917    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4e53b4-e8b6-4d43-96e4-877eea255369-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:24.088962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:24.088955    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6c6b\" (UniqueName: \"kubernetes.io/projected/1f4e53b4-e8b6-4d43-96e4-877eea255369-kube-api-access-b6c6b\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:24.295439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:24.295346    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:24.722210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:24.722184    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"
Oct 30 11:57:24.722402 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:24.722175    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs" event={"ID":"1f4e53b4-e8b6-4d43-96e4-877eea255369","Type":"ContainerDied","Data":"9eceef69989ffdf621698b4b1fa3220ecd2530a655eef9e45d38b3f8e0ce557b"}
Oct 30 11:57:24.722402 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:24.722322    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eceef69989ffdf621698b4b1fa3220ecd2530a655eef9e45d38b3f8e0ce557b"
Oct 30 11:57:26.738265 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:26.738209    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:28.503566 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:28.503524    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:30.765978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:30.765900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:32.657829 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.657784    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:32.789327 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.789287    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"]
Oct 30 11:57:32.790084 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.790054    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f4e53b4-e8b6-4d43-96e4-877eea255369" containerName="revoke-keys"
Oct 30 11:57:32.790084 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.790083    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4e53b4-e8b6-4d43-96e4-877eea255369" containerName="revoke-keys"
Oct 30 11:57:32.790302 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.790206    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4e53b4-e8b6-4d43-96e4-877eea255369" containerName="revoke-keys"
Oct 30 11:57:32.794248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.794219    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:32.812652 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.812397    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"]
Oct 30 11:57:32.826098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.825960    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:57:32.826098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.826016    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:57:32.826098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.826043    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:57:32.828772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.828744    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:57:32.934550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.934406    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:32.934741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.934629    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:32.934741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:32.934680    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.036211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.036169    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.036430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.036230    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.036430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.036384    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.037238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.037142    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.039113 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.039079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.060394 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.060353    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf\") pod \"maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.131261 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.131220    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:33.449395 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:57:33.449361    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac051f5_ff05_4819_aa40_cc7515a45ff3.slice/crio-b23673e7d0921e58cba86c5650d12c701689b2cc66886dcdb1f8bf0b161beee5 WatchSource:0}: Error finding container b23673e7d0921e58cba86c5650d12c701689b2cc66886dcdb1f8bf0b161beee5: Status 404 returned error can't find the container with id b23673e7d0921e58cba86c5650d12c701689b2cc66886dcdb1f8bf0b161beee5
Oct 30 11:57:33.450426 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.450359    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"]
Oct 30 11:57:33.805334 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:33.805292    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x" event={"ID":"4ac051f5-ff05-4819-aa40-cc7515a45ff3","Type":"ContainerStarted","Data":"b23673e7d0921e58cba86c5650d12c701689b2cc66886dcdb1f8bf0b161beee5"}
Oct 30 11:57:34.181758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.181669    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:34.431899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.431810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:34.675291 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.675255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:34.816383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.816353    5060 generic.go:356] "Generic (PLEG): container finished" podID="4ac051f5-ff05-4819-aa40-cc7515a45ff3" containerID="df449337b3d6da617d23219279db69bb8a09ff2f653c69028527f86255675629" exitCode=0
Oct 30 11:57:34.816878 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.816433    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x" event={"ID":"4ac051f5-ff05-4819-aa40-cc7515a45ff3","Type":"ContainerDied","Data":"df449337b3d6da617d23219279db69bb8a09ff2f653c69028527f86255675629"}
Oct 30 11:57:34.821246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.821217    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:34.986071 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:34.985969    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:35.189540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.171545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:35.336944 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.335855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:35.345576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.345531    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"]
Oct 30 11:57:35.349089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.349056    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c92679-znnbs"]
Oct 30 11:57:35.809986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.809950    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:35.825662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.825628    5060 generic.go:356] "Generic (PLEG): container finished" podID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerID="ebd2c0a3260fbb35afa0c7558f42f5485a9d9073f1e564a4ebce5f0cf7fd772e" exitCode=0
Oct 30 11:57:35.826137 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.825706    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" event={"ID":"ba0f9611-6af9-4819-b351-d8cad17c5866","Type":"ContainerDied","Data":"ebd2c0a3260fbb35afa0c7558f42f5485a9d9073f1e564a4ebce5f0cf7fd772e"}
Oct 30 11:57:35.826666 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.826278    5060 scope.go:122] "RemoveContainer" containerID="ebd2c0a3260fbb35afa0c7558f42f5485a9d9073f1e564a4ebce5f0cf7fd772e"
Oct 30 11:57:35.827945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.827908    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:57:35.828041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.827988    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:57:35.828041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.828020    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:57:35.927898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.926880    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"]
Oct 30 11:57:35.929277 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.929247    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8bb87-7x5fg"]
Oct 30 11:57:35.971798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:35.971759    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:36.078548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.075073    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:36.121997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.121642    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf\") pod \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") "
Oct 30 11:57:36.124864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.124339    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf" pod "4ac051f5-ff05-4819-aa40-cc7515a45ff3" (UID: "4ac051f5-ff05-4819-aa40-cc7515a45ff3"). InnerVolumeSpecName "kube-api-access-rxxjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:36.126712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.125927    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:36.222941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.222909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca\") pod \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") "
Oct 30 11:57:36.223112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.222958    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca\") pod \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\" (UID: \"4ac051f5-ff05-4819-aa40-cc7515a45ff3\") "
Oct 30 11:57:36.223325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.223302    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca" pod "4ac051f5-ff05-4819-aa40-cc7515a45ff3" (UID: "4ac051f5-ff05-4819-aa40-cc7515a45ff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:36.223512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.223491    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:36.223512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.223509    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxjf\" (UniqueName: \"kubernetes.io/projected/4ac051f5-ff05-4819-aa40-cc7515a45ff3-kube-api-access-rxxjf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:36.225278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.225246    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca" pod "4ac051f5-ff05-4819-aa40-cc7515a45ff3" (UID: "4ac051f5-ff05-4819-aa40-cc7515a45ff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:57:36.324977 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.324890    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac051f5-ff05-4819-aa40-cc7515a45ff3-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:36.335606 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.335531    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4e53b4-e8b6-4d43-96e4-877eea255369" path="/var/lib/kubelet/pods/1f4e53b4-e8b6-4d43-96e4-877eea255369/volumes"
Oct 30 11:57:36.335966 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.335936    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd759ddf-324a-47be-bb76-0c2b175c27b2" path="/var/lib/kubelet/pods/cd759ddf-324a-47be-bb76-0c2b175c27b2/volumes"
Oct 30 11:57:36.510174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.510137    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:36.836713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.836673    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" event={"ID":"ba0f9611-6af9-4819-b351-d8cad17c5866","Type":"ContainerStarted","Data":"1e4c7babd00909b1d403cab4d34c3a4ee98e3a8350bb80fc881483c9382343a7"}
Oct 30 11:57:36.838646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.838619    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"
Oct 30 11:57:36.838786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.838648    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x" event={"ID":"4ac051f5-ff05-4819-aa40-cc7515a45ff3","Type":"ContainerDied","Data":"b23673e7d0921e58cba86c5650d12c701689b2cc66886dcdb1f8bf0b161beee5"}
Oct 30 11:57:36.838786 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.838723    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23673e7d0921e58cba86c5650d12c701689b2cc66886dcdb1f8bf0b161beee5"
Oct 30 11:57:36.844091 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.844066    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:57:36.845551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.845516    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 11:57:36.845677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:36.845572    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 11:57:37.846839 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:37.844644    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 11:57:37.846839 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:37.844727    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 11:57:37.887700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:37.887663    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:57:38.537683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:38.537643    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:38.844856 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:38.844765    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 11:57:38.844856 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:38.844823    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 11:57:39.844448 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:39.844407    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 11:57:39.844896 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:39.844515    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 11:57:40.271474 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:40.270728    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:40.844735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:40.844691    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 11:57:40.845252 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:40.844752    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 11:57:41.739791 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:41.739698    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:41.834718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:41.834678    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"]
Oct 30 11:57:41.835419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:41.835359    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg" podUID="32033b38-f509-4a6c-a7e6-0e7b122376e2" containerName="payload-pre-processing" containerID="cri-o://82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653" gracePeriod=30
Oct 30 11:57:41.848979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:41.848940    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 11:57:41.849411 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:41.849017    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 11:57:41.946515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:41.940080    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:42.096490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.095262    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:42.344899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.344721    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:42.365028 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.363846    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:57:42.469489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.469413    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d\" (UniqueName: \"kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d\") pod \"32033b38-f509-4a6c-a7e6-0e7b122376e2\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") "
Oct 30 11:57:42.469694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.469538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume\") pod \"32033b38-f509-4a6c-a7e6-0e7b122376e2\" (UID: \"32033b38-f509-4a6c-a7e6-0e7b122376e2\") "
Oct 30 11:57:42.469930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.469885    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume" pod "32033b38-f509-4a6c-a7e6-0e7b122376e2" (UID: "32033b38-f509-4a6c-a7e6-0e7b122376e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:42.470112 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.470092    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32033b38-f509-4a6c-a7e6-0e7b122376e2-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:42.471905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.471884    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d" pod "32033b38-f509-4a6c-a7e6-0e7b122376e2" (UID: "32033b38-f509-4a6c-a7e6-0e7b122376e2"). InnerVolumeSpecName "kube-api-access-s658d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:42.486640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.486610    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"]
Oct 30 11:57:42.495815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.495791    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaee54-6f2sh"]
Oct 30 11:57:42.561781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.561694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:42.571152 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.571120    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s658d\" (UniqueName: \"kubernetes.io/projected/32033b38-f509-4a6c-a7e6-0e7b122376e2-kube-api-access-s658d\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:42.832724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.832684    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:42.849266 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.849232    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:57:42.850647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.850620    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:57:42.949231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.949193    5060 generic.go:356] "Generic (PLEG): container finished" podID="32033b38-f509-4a6c-a7e6-0e7b122376e2" containerID="82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653" exitCode=0
Oct 30 11:57:42.949388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.949246    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"
Oct 30 11:57:42.949388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.949271    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg" event={"ID":"32033b38-f509-4a6c-a7e6-0e7b122376e2","Type":"ContainerDied","Data":"82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653"}
Oct 30 11:57:42.949388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.949354    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg" event={"ID":"32033b38-f509-4a6c-a7e6-0e7b122376e2","Type":"ContainerDied","Data":"561b229f6d15ec941aab09f3b9df367da4841a45a5be096ef3a220ec04e64ea9"}
Oct 30 11:57:42.949388 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.949383    5060 scope.go:122] "RemoveContainer" containerID="82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653"
Oct 30 11:57:42.961384 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.961362    5060 scope.go:122] "RemoveContainer" containerID="82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653"
Oct 30 11:57:42.961748 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:57:42.961728    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653\": container with ID starting with 82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653 not found: ID does not exist" containerID="82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653"
Oct 30 11:57:42.961827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:42.961757    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653"} err="failed to get container status \"82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653\": rpc error: code = NotFound desc = could not find container \"82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653\": container with ID starting with 82d0d69460bd3a2580bfb55dd90965a72d5ef82c62c39a24b64b97b551f80653 not found: ID does not exist"
Oct 30 11:57:43.022840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.022805    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"]
Oct 30 11:57:43.039413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.039367    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:43.039624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.039603    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaee54-76cd495699-dfzlg"]
Oct 30 11:57:43.238274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.238240    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:43.614723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.614640    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"]
Oct 30 11:57:43.615230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.615216    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4ac051f5-ff05-4819-aa40-cc7515a45ff3" containerName="revoke-keys"
Oct 30 11:57:43.615273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.615232    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac051f5-ff05-4819-aa40-cc7515a45ff3" containerName="revoke-keys"
Oct 30 11:57:43.615273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.615255    5060 cpu_manager.go:414] "containerMap: removing container" podUID="32033b38-f509-4a6c-a7e6-0e7b122376e2" containerName="payload-pre-processing"
Oct 30 11:57:43.615273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.615261    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32033b38-f509-4a6c-a7e6-0e7b122376e2" containerName="payload-pre-processing"
Oct 30 11:57:43.615373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.615347    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="32033b38-f509-4a6c-a7e6-0e7b122376e2" containerName="payload-pre-processing"
Oct 30 11:57:43.615373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.615362    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac051f5-ff05-4819-aa40-cc7515a45ff3" containerName="revoke-keys"
Oct 30 11:57:43.620555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.620522    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.637039 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.637004    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"]
Oct 30 11:57:43.645209 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.645177    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:57:43.645209 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.645187    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:57:43.652680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.652650    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:57:43.652680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.652650    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:57:43.708744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.708711    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.708971 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.708940    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.709213 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.709030    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.809930 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.809892    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.810140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.809960    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.810140 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.810037    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.810684 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.810658    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.812638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.812614    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.834178 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.834143    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh\") pod \"maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:43.937114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:43.937023    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:44.249742 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:57:44.249705    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc73e05_c3d6_4239_843e_84d197cd7e58.slice/crio-ba362ef76b2d7d8e4c745ac6e1b047ee1a19f5aaf21d46d7d06c7e36d23c9c94 WatchSource:0}: Error finding container ba362ef76b2d7d8e4c745ac6e1b047ee1a19f5aaf21d46d7d06c7e36d23c9c94: Status 404 returned error can't find the container with id ba362ef76b2d7d8e4c745ac6e1b047ee1a19f5aaf21d46d7d06c7e36d23c9c94
Oct 30 11:57:44.252068 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.252040    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"]
Oct 30 11:57:44.336260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.336222    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32033b38-f509-4a6c-a7e6-0e7b122376e2" path="/var/lib/kubelet/pods/32033b38-f509-4a6c-a7e6-0e7b122376e2/volumes"
Oct 30 11:57:44.336713 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.336699    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bba782-5bbc-469a-b020-afbb4e485c93" path="/var/lib/kubelet/pods/f9bba782-5bbc-469a-b020-afbb4e485c93/volumes"
Oct 30 11:57:44.731873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.731829    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:44.899989 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.899895    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:44.969690 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.969654    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt" event={"ID":"1bc73e05-c3d6-4239-843e-84d197cd7e58","Type":"ContainerStarted","Data":"2695d1a44ab711720d2b5c16d23015067b7e10ef4f2f65802429fa32071300e7"}
Oct 30 11:57:44.970163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:44.969716    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt" event={"ID":"1bc73e05-c3d6-4239-843e-84d197cd7e58","Type":"ContainerStarted","Data":"ba362ef76b2d7d8e4c745ac6e1b047ee1a19f5aaf21d46d7d06c7e36d23c9c94"}
Oct 30 11:57:45.016614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.016554    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt" podStartSLOduration=2.016539831 podStartE2EDuration="2.016539831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:45.015479974 +0000 UTC m=+1668.810364471" watchObservedRunningTime="2026-10-30 11:57:45.016539831 +0000 UTC m=+1668.811424325"
Oct 30 11:57:45.170707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.170608    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"]
Oct 30 11:57:45.170900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.170859    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy" containerID="cri-o://1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6" gracePeriod=30
Oct 30 11:57:45.255772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.255741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:45.805967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.805925    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"]
Oct 30 11:57:45.814703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.814676    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:45.815004 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.814976    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"]
Oct 30 11:57:45.939512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.939465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:45.939733 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.939639    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:45.939733 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.939707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:45.980019 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.979981    5060 generic.go:356] "Generic (PLEG): container finished" podID="1bc73e05-c3d6-4239-843e-84d197cd7e58" containerID="2695d1a44ab711720d2b5c16d23015067b7e10ef4f2f65802429fa32071300e7" exitCode=0
Oct 30 11:57:45.980430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:45.980072    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt" event={"ID":"1bc73e05-c3d6-4239-843e-84d197cd7e58","Type":"ContainerDied","Data":"2695d1a44ab711720d2b5c16d23015067b7e10ef4f2f65802429fa32071300e7"}
Oct 30 11:57:46.041577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.041532    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.041793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.041627    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.041793 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.041783    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.042384 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.042357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.044675 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.044651    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.066101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.066020    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7\") pod \"maas-api-revoke-keys-e2e-api-b-73dffe-46kj2\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.141419 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.141378    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:46.463694 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:57:46.463665    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98404b44_dddd_4671_8443_febd85319963.slice/crio-5f14a4021611dc9f88656e1db315a3c4e47d6ded6d2e7a291fb50246f7ac5f60 WatchSource:0}: Error finding container 5f14a4021611dc9f88656e1db315a3c4e47d6ded6d2e7a291fb50246f7ac5f60: Status 404 returned error can't find the container with id 5f14a4021611dc9f88656e1db315a3c4e47d6ded6d2e7a291fb50246f7ac5f60
Oct 30 11:57:46.464874 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.464848    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"]
Oct 30 11:57:46.905872 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.905838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:46.992407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.992368    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2" event={"ID":"98404b44-dddd-4671-8443-febd85319963","Type":"ContainerStarted","Data":"c9857aeca64e073e5367d2e13c718438cab96a610591216cf92075ca89bfbaa8"}
Oct 30 11:57:46.992942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:46.992433    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2" event={"ID":"98404b44-dddd-4671-8443-febd85319963","Type":"ContainerStarted","Data":"5f14a4021611dc9f88656e1db315a3c4e47d6ded6d2e7a291fb50246f7ac5f60"}
Oct 30 11:57:47.033834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.033781    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2" podStartSLOduration=2.033768186 podStartE2EDuration="2.033768186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:47.033039649 +0000 UTC m=+1670.827924137" watchObservedRunningTime="2026-10-30 11:57:47.033768186 +0000 UTC m=+1670.828652684"
Oct 30 11:57:47.167005 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.166977    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:47.256824 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.256792    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh\") pod \"1bc73e05-c3d6-4239-843e-84d197cd7e58\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") "
Oct 30 11:57:47.257000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.256987    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca\") pod \"1bc73e05-c3d6-4239-843e-84d197cd7e58\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") "
Oct 30 11:57:47.257080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.257048    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca\") pod \"1bc73e05-c3d6-4239-843e-84d197cd7e58\" (UID: \"1bc73e05-c3d6-4239-843e-84d197cd7e58\") "
Oct 30 11:57:47.257433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.257400    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca" pod "1bc73e05-c3d6-4239-843e-84d197cd7e58" (UID: "1bc73e05-c3d6-4239-843e-84d197cd7e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:47.259279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.259251    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh" pod "1bc73e05-c3d6-4239-843e-84d197cd7e58" (UID: "1bc73e05-c3d6-4239-843e-84d197cd7e58"). InnerVolumeSpecName "kube-api-access-zq9jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:47.259378 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.259324    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca" pod "1bc73e05-c3d6-4239-843e-84d197cd7e58" (UID: "1bc73e05-c3d6-4239-843e-84d197cd7e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:57:47.359093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.359012    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:47.359093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.359041    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc73e05-c3d6-4239-843e-84d197cd7e58-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:47.359093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:47.359050    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/1bc73e05-c3d6-4239-843e-84d197cd7e58-kube-api-access-zq9jh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:48.002076 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:48.002043    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"
Oct 30 11:57:48.002618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:48.002087    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt" event={"ID":"1bc73e05-c3d6-4239-843e-84d197cd7e58","Type":"ContainerDied","Data":"ba362ef76b2d7d8e4c745ac6e1b047ee1a19f5aaf21d46d7d06c7e36d23c9c94"}
Oct 30 11:57:48.002618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:48.002151    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba362ef76b2d7d8e4c745ac6e1b047ee1a19f5aaf21d46d7d06c7e36d23c9c94"
Oct 30 11:57:48.004165 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:48.004134    5060 generic.go:356] "Generic (PLEG): container finished" podID="98404b44-dddd-4671-8443-febd85319963" containerID="c9857aeca64e073e5367d2e13c718438cab96a610591216cf92075ca89bfbaa8" exitCode=0
Oct 30 11:57:48.004306 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:48.004249    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2" event={"ID":"98404b44-dddd-4671-8443-febd85319963","Type":"ContainerDied","Data":"c9857aeca64e073e5367d2e13c718438cab96a610591216cf92075ca89bfbaa8"}
Oct 30 11:57:48.787025 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:48.786986    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:49.187546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.187507    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:49.293724 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.293674    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7\") pod \"98404b44-dddd-4671-8443-febd85319963\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") "
Oct 30 11:57:49.293940 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.293794    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca\") pod \"98404b44-dddd-4671-8443-febd85319963\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") "
Oct 30 11:57:49.293940 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.293881    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca\") pod \"98404b44-dddd-4671-8443-febd85319963\" (UID: \"98404b44-dddd-4671-8443-febd85319963\") "
Oct 30 11:57:49.294268 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.294242    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca" pod "98404b44-dddd-4671-8443-febd85319963" (UID: "98404b44-dddd-4671-8443-febd85319963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:49.294471 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.294437    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98404b44-dddd-4671-8443-febd85319963-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:49.295998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.295962    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca" pod "98404b44-dddd-4671-8443-febd85319963" (UID: "98404b44-dddd-4671-8443-febd85319963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:57:49.296125 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.296027    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7" pod "98404b44-dddd-4671-8443-febd85319963" (UID: "98404b44-dddd-4671-8443-febd85319963"). InnerVolumeSpecName "kube-api-access-wh4j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:49.395782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.395684    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4j7\" (UniqueName: \"kubernetes.io/projected/98404b44-dddd-4671-8443-febd85319963-kube-api-access-wh4j7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:49.395782 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.395722    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98404b44-dddd-4671-8443-febd85319963-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:49.720029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.719983    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:49.877959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:49.877926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:49.950830 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:57:49.950787    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:57:50.024360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.024279    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"
Oct 30 11:57:50.024360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.024304    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2" event={"ID":"98404b44-dddd-4671-8443-febd85319963","Type":"ContainerDied","Data":"5f14a4021611dc9f88656e1db315a3c4e47d6ded6d2e7a291fb50246f7ac5f60"}
Oct 30 11:57:50.024360 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.024351    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f14a4021611dc9f88656e1db315a3c4e47d6ded6d2e7a291fb50246f7ac5f60"
Oct 30 11:57:50.133390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.133351    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:50.366891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.366800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:50.517066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.517042    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:57:50.558065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558033    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558082    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558115    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558147    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558216    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558252    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558324    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558366    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:57:50.558519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558408    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert\") pod \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\" (UID: \"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa\") "
Oct 30 11:57:50.558792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:57:50.558900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558871    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:57:50.559134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.558940    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:57:50.559134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.559094    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:57:50.559250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.559142    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.559250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.559158    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.559250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.559169    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.559250 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.559178    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.560642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.560613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:57:50.560752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.560639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:50.561007 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.560992    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:57:50.561084 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.561064    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67" pod "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" (UID: "d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa"). InnerVolumeSpecName "kube-api-access-scx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:57:50.660708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.660629    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.660708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.660656    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.660708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.660665    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.660708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.660673    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:50.660708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:50.660683    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scx67\" (UniqueName: \"kubernetes.io/projected/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa-kube-api-access-scx67\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:57:51.034527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.034485    5060 generic.go:356] "Generic (PLEG): container finished" podID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerID="1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6" exitCode=0
Oct 30 11:57:51.034721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.034533    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"
Oct 30 11:57:51.034721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.034538    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" event={"ID":"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa","Type":"ContainerDied","Data":"1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6"}
Oct 30 11:57:51.034721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.034599    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98" event={"ID":"d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa","Type":"ContainerDied","Data":"eafa5b4ffc0f9f0c87ebb703603371933b43afbcd4f2834edf18777e9fcd30c0"}
Oct 30 11:57:51.034721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.034621    5060 scope.go:122] "RemoveContainer" containerID="1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6"
Oct 30 11:57:51.047225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.047200    5060 scope.go:122] "RemoveContainer" containerID="33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"
Oct 30 11:57:51.057547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.057528    5060 scope.go:122] "RemoveContainer" containerID="1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6"
Oct 30 11:57:51.057846 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:57:51.057826    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6\": container with ID starting with 1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6 not found: ID does not exist" containerID="1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6"
Oct 30 11:57:51.057913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.057855    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6"} err="failed to get container status \"1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6\": rpc error: code = NotFound desc = could not find container \"1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6\": container with ID starting with 1819d93ae1f717dd7d035257c09a49c398aad6d5c849973f05797939702207b6 not found: ID does not exist"
Oct 30 11:57:51.057913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.057876    5060 scope.go:122] "RemoveContainer" containerID="33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"
Oct 30 11:57:51.058121 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:57:51.058103    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484\": container with ID starting with 33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484 not found: ID does not exist" containerID="33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"
Oct 30 11:57:51.058184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.058131    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484"} err="failed to get container status \"33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484\": rpc error: code = NotFound desc = could not find container \"33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484\": container with ID starting with 33c6d6b43a470547977b2bf086cc73c986ff76f5b346a46638575197b4bdc484 not found: ID does not exist"
Oct 30 11:57:51.094376 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.094340    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"]
Oct 30 11:57:51.107034 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.107002    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c8bb87-openshift-default-7658ccf69c-thg98"]
Oct 30 11:57:51.407970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:51.407807    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:52.336313 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:52.336279    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" path="/var/lib/kubelet/pods/d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa/volumes"
Oct 30 11:57:53.174512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:53.174439    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:55.460056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:55.460017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:56.337409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:56.337378    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"]
Oct 30 11:57:56.373259 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:56.373227    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"]
Oct 30 11:57:56.835251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:56.835215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:57.246080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:57.246036    5060 scope.go:122] "RemoveContainer" containerID="694c32e6ae373675b8013fd192b6db825c5b7c71adc8610c2e1fd2e4a2a6e690"
Oct 30 11:57:57.259519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:57.259492    5060 scope.go:122] "RemoveContainer" containerID="934355da24bf16dfca8152b50361c5a68db437596d8bee2aeb775f304feb9567"
Oct 30 11:57:57.274777 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:57.274753    5060 scope.go:122] "RemoveContainer" containerID="7768f10a971db5e142345388e08b41945ae14555f10f625c0d601b49c1860481"
Oct 30 11:57:57.532864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:57.532768    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:59.064258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064221    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"]
Oct 30 11:57:59.064816 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064800    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy"
Oct 30 11:57:59.064861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064817    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy"
Oct 30 11:57:59.064861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064832    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy"
Oct 30 11:57:59.064861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064838    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy"
Oct 30 11:57:59.064962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064862    5060 cpu_manager.go:414] "containerMap: removing container" podUID="98404b44-dddd-4671-8443-febd85319963" containerName="revoke-keys"
Oct 30 11:57:59.064962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064868    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98404b44-dddd-4671-8443-febd85319963" containerName="revoke-keys"
Oct 30 11:57:59.064962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064884    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1bc73e05-c3d6-4239-843e-84d197cd7e58" containerName="revoke-keys"
Oct 30 11:57:59.064962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064888    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc73e05-c3d6-4239-843e-84d197cd7e58" containerName="revoke-keys"
Oct 30 11:57:59.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064968    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="98404b44-dddd-4671-8443-febd85319963" containerName="revoke-keys"
Oct 30 11:57:59.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064979    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc73e05-c3d6-4239-843e-84d197cd7e58" containerName="revoke-keys"
Oct 30 11:57:59.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064989    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy"
Oct 30 11:57:59.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.064994    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66e7a5d-6f51-4d87-a0fe-0dcfba8deefa" containerName="istio-proxy"
Oct 30 11:57:59.070135 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.070115    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.086320 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.086285    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"]
Oct 30 11:57:59.096254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.094302    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:57:59.096254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.094348    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:57:59.096254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.094372    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:57:59.097475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.097433    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:57:59.187244 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.187210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.187425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.187398    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.187524 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.187490    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l2f\" (UniqueName: \"kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.289205 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.289173    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.289414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.289312    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.289414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.289374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l2f\" (UniqueName: \"kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.290078 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.290003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.291906 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.291885    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.313960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.313930    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l2f\" (UniqueName: \"kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f\") pod \"maas-api-revoke-sub-8badc062-mlqcp\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.390834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.390735    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:57:59.435439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.435398    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:59.606517 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.606256    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:57:59.708924 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:57:59.708885    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7031cfb_2a81_4221_9912_2fde25a09226.slice/crio-72c6cda5001e2490699c718771d8e7fa9d24647afa0b99c44874836cbab416b8 WatchSource:0}: Error finding container 72c6cda5001e2490699c718771d8e7fa9d24647afa0b99c44874836cbab416b8: Status 404 returned error can't find the container with id 72c6cda5001e2490699c718771d8e7fa9d24647afa0b99c44874836cbab416b8
Oct 30 11:57:59.712149 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:57:59.712111    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"]
Oct 30 11:58:00.133844 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:00.133805    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp" event={"ID":"a7031cfb-2a81-4221-9912-2fde25a09226","Type":"ContainerStarted","Data":"b05469dcded8de5801dd54814628e670feb9893cbe29e0c070ff36e89df1dbf4"}
Oct 30 11:58:00.134223 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:00.133867    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp" event={"ID":"a7031cfb-2a81-4221-9912-2fde25a09226","Type":"ContainerStarted","Data":"72c6cda5001e2490699c718771d8e7fa9d24647afa0b99c44874836cbab416b8"}
Oct 30 11:58:00.204892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:00.204833    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp" podStartSLOduration=1.204816468 podStartE2EDuration="1.204816468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:00.19564984 +0000 UTC m=+1683.990534323" watchObservedRunningTime="2026-10-30 11:58:00.204816468 +0000 UTC m=+1683.999700963"
Oct 30 11:58:00.852848 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:00.852813    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:01.143128 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.143095    5060 generic.go:356] "Generic (PLEG): container finished" podID="a7031cfb-2a81-4221-9912-2fde25a09226" containerID="b05469dcded8de5801dd54814628e670feb9893cbe29e0c070ff36e89df1dbf4" exitCode=0
Oct 30 11:58:01.143536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.143257    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp" event={"ID":"a7031cfb-2a81-4221-9912-2fde25a09226","Type":"ContainerDied","Data":"b05469dcded8de5801dd54814628e670feb9893cbe29e0c070ff36e89df1dbf4"}
Oct 30 11:58:01.156494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.156447    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:01.337731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.337682    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5" podUID="97cb61db-69bd-4634-b900-aca2c172aa47" containerName="payload-processing" containerID="cri-o://883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed" gracePeriod=25
Oct 30 11:58:01.338009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.337981    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:01.374607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.374555    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl" podUID="3c0a9518-4c8c-47f1-94b9-716cdb26004f" containerName="payload-pre-processing" containerID="cri-o://eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3" gracePeriod=25
Oct 30 11:58:01.739570 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.733760    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:01.775327 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.775291    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:58:01.834998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.834730    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:58:01.913093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.913056    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:01.955065 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955036    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx\" (UniqueName: \"kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx\") pod \"97cb61db-69bd-4634-b900-aca2c172aa47\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") "
Oct 30 11:58:01.955256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955150    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume\") pod \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") "
Oct 30 11:58:01.955256 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955182    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96\" (UniqueName: \"kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96\") pod \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\" (UID: \"3c0a9518-4c8c-47f1-94b9-716cdb26004f\") "
Oct 30 11:58:01.955374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955353    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume\") pod \"97cb61db-69bd-4634-b900-aca2c172aa47\" (UID: \"97cb61db-69bd-4634-b900-aca2c172aa47\") "
Oct 30 11:58:01.955581 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955556    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume" pod "3c0a9518-4c8c-47f1-94b9-716cdb26004f" (UID: "3c0a9518-4c8c-47f1-94b9-716cdb26004f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:01.955775 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955753    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume" pod "97cb61db-69bd-4634-b900-aca2c172aa47" (UID: "97cb61db-69bd-4634-b900-aca2c172aa47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:01.955913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955896    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cb61db-69bd-4634-b900-aca2c172aa47-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:01.955969 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.955919    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a9518-4c8c-47f1-94b9-716cdb26004f-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:01.957531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.957510    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx" pod "97cb61db-69bd-4634-b900-aca2c172aa47" (UID: "97cb61db-69bd-4634-b900-aca2c172aa47"). InnerVolumeSpecName "kube-api-access-r6wdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:01.957579 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:01.957542    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96" pod "3c0a9518-4c8c-47f1-94b9-716cdb26004f" (UID: "3c0a9518-4c8c-47f1-94b9-716cdb26004f"). InnerVolumeSpecName "kube-api-access-zxg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:02.057098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.057007    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6wdx\" (UniqueName: \"kubernetes.io/projected/97cb61db-69bd-4634-b900-aca2c172aa47-kube-api-access-r6wdx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:02.057098 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.057050    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxg96\" (UniqueName: \"kubernetes.io/projected/3c0a9518-4c8c-47f1-94b9-716cdb26004f-kube-api-access-zxg96\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:02.064309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.064273    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:02.152619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.152576    5060 generic.go:356] "Generic (PLEG): container finished" podID="97cb61db-69bd-4634-b900-aca2c172aa47" containerID="883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed" exitCode=0
Oct 30 11:58:02.153093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.152646    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"
Oct 30 11:58:02.153093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.152669    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5" event={"ID":"97cb61db-69bd-4634-b900-aca2c172aa47","Type":"ContainerDied","Data":"883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed"}
Oct 30 11:58:02.153093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.152729    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5" event={"ID":"97cb61db-69bd-4634-b900-aca2c172aa47","Type":"ContainerDied","Data":"288d484ee8dafaa35ddb74cd14af09f3aebbba958dbe3f3116a9faa5f984b371"}
Oct 30 11:58:02.153093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.152753    5060 scope.go:122] "RemoveContainer" containerID="883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed"
Oct 30 11:58:02.154596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.154569    5060 generic.go:356] "Generic (PLEG): container finished" podID="3c0a9518-4c8c-47f1-94b9-716cdb26004f" containerID="eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3" exitCode=0
Oct 30 11:58:02.154731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.154642    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"
Oct 30 11:58:02.154731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.154672    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl" event={"ID":"3c0a9518-4c8c-47f1-94b9-716cdb26004f","Type":"ContainerDied","Data":"eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3"}
Oct 30 11:58:02.154731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.154727    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl" event={"ID":"3c0a9518-4c8c-47f1-94b9-716cdb26004f","Type":"ContainerDied","Data":"b9659c23b36db16787ce11358956f852176d82733bf136d60ea2ac2921756859"}
Oct 30 11:58:02.186480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.186181    5060 scope.go:122] "RemoveContainer" containerID="883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed"
Oct 30 11:58:02.195978 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:02.194119    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed\": container with ID starting with 883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed not found: ID does not exist" containerID="883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed"
Oct 30 11:58:02.195978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.194184    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed"} err="failed to get container status \"883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed\": rpc error: code = NotFound desc = could not find container \"883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed\": container with ID starting with 883aaf5c9d1a02ba086ef97281c5ff8ec0b8d03fd488f751868b82bf45bdaaed not found: ID does not exist"
Oct 30 11:58:02.195978 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.194214    5060 scope.go:122] "RemoveContainer" containerID="eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3"
Oct 30 11:58:02.257008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.256505    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"]
Oct 30 11:58:02.268396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.264141    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73dffe-84d65777b6-rvcg5"]
Oct 30 11:58:02.280493 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.272658    5060 scope.go:122] "RemoveContainer" containerID="eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3"
Oct 30 11:58:02.289362 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:02.288780    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3\": container with ID starting with eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3 not found: ID does not exist" containerID="eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3"
Oct 30 11:58:02.289362 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.288835    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3"} err="failed to get container status \"eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3\": rpc error: code = NotFound desc = could not find container \"eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3\": container with ID starting with eeb23a874251eaeb772a8deb075d0bda62c42f0a78de54bf3f28fbf110eef7d3 not found: ID does not exist"
Oct 30 11:58:02.290539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.290496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:02.338298 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.338114    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cb61db-69bd-4634-b900-aca2c172aa47" path="/var/lib/kubelet/pods/97cb61db-69bd-4634-b900-aca2c172aa47/volumes"
Oct 30 11:58:02.338585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.338565    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"]
Oct 30 11:58:02.342116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.342070    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73dffe-7b46dbc6ff-7qlvl"]
Oct 30 11:58:02.463340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.460058    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:02.504477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.488223    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:58:02.576134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.576104    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f\" (UniqueName: \"kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f\") pod \"a7031cfb-2a81-4221-9912-2fde25a09226\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") "
Oct 30 11:58:02.576347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.576170    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca\") pod \"a7031cfb-2a81-4221-9912-2fde25a09226\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") "
Oct 30 11:58:02.576608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.576578    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca\") pod \"a7031cfb-2a81-4221-9912-2fde25a09226\" (UID: \"a7031cfb-2a81-4221-9912-2fde25a09226\") "
Oct 30 11:58:02.577224 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.577202    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca" pod "a7031cfb-2a81-4221-9912-2fde25a09226" (UID: "a7031cfb-2a81-4221-9912-2fde25a09226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:02.583141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.583056    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca" pod "a7031cfb-2a81-4221-9912-2fde25a09226" (UID: "a7031cfb-2a81-4221-9912-2fde25a09226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:02.583141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.583094    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f" pod "a7031cfb-2a81-4221-9912-2fde25a09226" (UID: "a7031cfb-2a81-4221-9912-2fde25a09226"). InnerVolumeSpecName "kube-api-access-65l2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:02.677735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.677697    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l2f\" (UniqueName: \"kubernetes.io/projected/a7031cfb-2a81-4221-9912-2fde25a09226-kube-api-access-65l2f\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:02.677735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.677727    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:02.677735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:02.677740    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7031cfb-2a81-4221-9912-2fde25a09226-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:03.166375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:03.166340    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"
Oct 30 11:58:03.166375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:03.166361    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp" event={"ID":"a7031cfb-2a81-4221-9912-2fde25a09226","Type":"ContainerDied","Data":"72c6cda5001e2490699c718771d8e7fa9d24647afa0b99c44874836cbab416b8"}
Oct 30 11:58:03.166996 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:03.166421    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72c6cda5001e2490699c718771d8e7fa9d24647afa0b99c44874836cbab416b8"
Oct 30 11:58:03.747554 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:03.747519    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:04.335351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:04.335316    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0a9518-4c8c-47f1-94b9-716cdb26004f" path="/var/lib/kubelet/pods/3c0a9518-4c8c-47f1-94b9-716cdb26004f/volumes"
Oct 30 11:58:05.666641 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:05.666608    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:06.352188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:06.352140    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:06.512650 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:06.512614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:07.780492 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:07.780383    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:08.156228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.156136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:08.294131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294093    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"]
Oct 30 11:58:08.294670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294655    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a7031cfb-2a81-4221-9912-2fde25a09226" containerName="revoke-keys"
Oct 30 11:58:08.294670 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294670    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7031cfb-2a81-4221-9912-2fde25a09226" containerName="revoke-keys"
Oct 30 11:58:08.294817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294687    5060 cpu_manager.go:414] "containerMap: removing container" podUID="97cb61db-69bd-4634-b900-aca2c172aa47" containerName="payload-processing"
Oct 30 11:58:08.294817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294708    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cb61db-69bd-4634-b900-aca2c172aa47" containerName="payload-processing"
Oct 30 11:58:08.294817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294721    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3c0a9518-4c8c-47f1-94b9-716cdb26004f" containerName="payload-pre-processing"
Oct 30 11:58:08.294817 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294726    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0a9518-4c8c-47f1-94b9-716cdb26004f" containerName="payload-pre-processing"
Oct 30 11:58:08.295026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294834    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cb61db-69bd-4634-b900-aca2c172aa47" containerName="payload-processing"
Oct 30 11:58:08.295026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294843    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0a9518-4c8c-47f1-94b9-716cdb26004f" containerName="payload-pre-processing"
Oct 30 11:58:08.295026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.294857    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7031cfb-2a81-4221-9912-2fde25a09226" containerName="revoke-keys"
Oct 30 11:58:08.299636 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.299610    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.317556 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.317523    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"]
Oct 30 11:58:08.324756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.324721    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6fcda5\""
Oct 30 11:58:08.324756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.324723    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6fcda5-dockercfg-r86cj\""
Oct 30 11:58:08.347781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.347737    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:08.361351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.361313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdpn\" (UniqueName: \"kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn\") pod \"payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.361836 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.361650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.502480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.502422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.502692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.502606    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdpn\" (UniqueName: \"kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn\") pod \"payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.503278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.503252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.530632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.530598    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdpn\" (UniqueName: \"kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn\") pod \"payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.619200 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.619164    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:08.930156 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:08.930125    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d0a6ed_db94_4ce0_9bad_6be745ff99e1.slice/crio-0c05e39d3dd464e26d7f34b18dc6ca964453f188e97facc442b1e698d226fd57 WatchSource:0}: Error finding container 0c05e39d3dd464e26d7f34b18dc6ca964453f188e97facc442b1e698d226fd57: Status 404 returned error can't find the container with id 0c05e39d3dd464e26d7f34b18dc6ca964453f188e97facc442b1e698d226fd57
Oct 30 11:58:08.933374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:08.933350    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"]
Oct 30 11:58:09.059383 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:09.059347    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"]
Oct 30 11:58:09.071516 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:09.071480    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f932b170-b899m"]
Oct 30 11:58:09.219066 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:09.219022    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm" event={"ID":"26d0a6ed-db94-4ce0-9bad-6be745ff99e1","Type":"ContainerStarted","Data":"dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b"}
Oct 30 11:58:09.219283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:09.219081    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm" event={"ID":"26d0a6ed-db94-4ce0-9bad-6be745ff99e1","Type":"ContainerStarted","Data":"0c05e39d3dd464e26d7f34b18dc6ca964453f188e97facc442b1e698d226fd57"}
Oct 30 11:58:09.219283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:09.219107    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:09.270533 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:09.270432    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm" podStartSLOduration=1.270413418 podStartE2EDuration="1.270413418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:09.268216822 +0000 UTC m=+1693.063101319" watchObservedRunningTime="2026-10-30 11:58:09.270413418 +0000 UTC m=+1693.065297913"
Oct 30 11:58:10.335009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:10.334977    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f1d2a3-b390-4a74-9391-164f622a2378" path="/var/lib/kubelet/pods/e8f1d2a3-b390-4a74-9391-164f622a2378/volumes"
Oct 30 11:58:10.447465 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:10.447403    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:11.062796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:11.062762    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"]
Oct 30 11:58:11.062993 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:11.062959    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy" containerID="cri-o://a63a7dc87ffe3dbaef62234702bbbf27d0501c2665e679dba0b0842c9ec0f243" gracePeriod=30
Oct 30 11:58:12.081425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.081386    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:12.172402 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.172357    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"]
Oct 30 11:58:12.172691 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.172659    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4" podUID="94aefcd2-20db-4f74-8287-6e40074b270f" containerName="payload-pre-processing" containerID="cri-o://31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26" gracePeriod=30
Oct 30 11:58:12.267415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.267367    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:12.413392 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.413354    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:12.617046 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.615832    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:58:12.662198 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.662167    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume\") pod \"94aefcd2-20db-4f74-8287-6e40074b270f\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") "
Oct 30 11:58:12.662399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.662304    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59\" (UniqueName: \"kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59\") pod \"94aefcd2-20db-4f74-8287-6e40074b270f\" (UID: \"94aefcd2-20db-4f74-8287-6e40074b270f\") "
Oct 30 11:58:12.662691 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.662668    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume" pod "94aefcd2-20db-4f74-8287-6e40074b270f" (UID: "94aefcd2-20db-4f74-8287-6e40074b270f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:12.662998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.662977    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aefcd2-20db-4f74-8287-6e40074b270f-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:12.665293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.665263    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59" pod "94aefcd2-20db-4f74-8287-6e40074b270f" (UID: "94aefcd2-20db-4f74-8287-6e40074b270f"). InnerVolumeSpecName "kube-api-access-h4v59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:12.705324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.705272    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:12.764175 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.764004    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4v59\" (UniqueName: \"kubernetes.io/projected/94aefcd2-20db-4f74-8287-6e40074b270f-kube-api-access-h4v59\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:12.826210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.826173    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"]
Oct 30 11:58:12.830255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.830225    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648839-5d6jt"]
Oct 30 11:58:12.875397 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:12.875287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:13.255972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.255938    5060 generic.go:356] "Generic (PLEG): container finished" podID="94aefcd2-20db-4f74-8287-6e40074b270f" containerID="31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26" exitCode=0
Oct 30 11:58:13.256374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.255986    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"
Oct 30 11:58:13.256374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.256029    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4" event={"ID":"94aefcd2-20db-4f74-8287-6e40074b270f","Type":"ContainerDied","Data":"31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26"}
Oct 30 11:58:13.256374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.256081    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4" event={"ID":"94aefcd2-20db-4f74-8287-6e40074b270f","Type":"ContainerDied","Data":"66423e80c4d5e90e68011df165a5ab5fc697c31884adb1f6481136cf0c14eb8c"}
Oct 30 11:58:13.256374 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.256107    5060 scope.go:122] "RemoveContainer" containerID="31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26"
Oct 30 11:58:13.267596 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.267575    5060 scope.go:122] "RemoveContainer" containerID="31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26"
Oct 30 11:58:13.267959 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:13.267930    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26\": container with ID starting with 31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26 not found: ID does not exist" containerID="31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26"
Oct 30 11:58:13.268041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.267968    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26"} err="failed to get container status \"31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26\": rpc error: code = NotFound desc = could not find container \"31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26\": container with ID starting with 31f863b8b08f141f974e94b84b0f50899875e7694011948954ae3783b6ff5e26 not found: ID does not exist"
Oct 30 11:58:13.312939 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.312908    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"]
Oct 30 11:58:13.325258 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.325225    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648839-75cf86f87f-fgzz4"]
Oct 30 11:58:13.439909 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:13.439877    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:14.344569 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:14.344536    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc73e05-c3d6-4239-843e-84d197cd7e58" path="/var/lib/kubelet/pods/1bc73e05-c3d6-4239-843e-84d197cd7e58/volumes"
Oct 30 11:58:14.345000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:14.344985    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94aefcd2-20db-4f74-8287-6e40074b270f" path="/var/lib/kubelet/pods/94aefcd2-20db-4f74-8287-6e40074b270f/volumes"
Oct 30 11:58:14.972187 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:14.972150    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:16.295505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.293987    5060 generic.go:356] "Generic (PLEG): container finished" podID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerID="a63a7dc87ffe3dbaef62234702bbbf27d0501c2665e679dba0b0842c9ec0f243" exitCode=0
Oct 30 11:58:16.295505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.294117    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" event={"ID":"3e8c3306-98ff-4d83-a576-78aba00c24ee","Type":"ContainerDied","Data":"a63a7dc87ffe3dbaef62234702bbbf27d0501c2665e679dba0b0842c9ec0f243"}
Oct 30 11:58:16.351416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.351390    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:58:16.416960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.416925    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.417236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417214    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.417437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417420    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.417649 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417629    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.417792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417775    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.417947 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417932    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.418089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.418074    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.418230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.418213    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.418423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.418408    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs\") pod \"3e8c3306-98ff-4d83-a576-78aba00c24ee\" (UID: \"3e8c3306-98ff-4d83-a576-78aba00c24ee\") "
Oct 30 11:58:16.419323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:16.419323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.417847    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:16.419523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.418000    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:16.419523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.418256    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:16.419700 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.419673    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:16.424618 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.424559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:16.425202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.425165    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "kube-api-access-xpbdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:16.425733 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.425697    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:16.427126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.427098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy" pod "3e8c3306-98ff-4d83-a576-78aba00c24ee" (UID: "3e8c3306-98ff-4d83-a576-78aba00c24ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:16.521001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.520943    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.520999    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521017    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521032    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521046    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8c3306-98ff-4d83-a576-78aba00c24ee-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521060    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbdt\" (UniqueName: \"kubernetes.io/projected/3e8c3306-98ff-4d83-a576-78aba00c24ee-kube-api-access-xpbdt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521072    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521085    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8c3306-98ff-4d83-a576-78aba00c24ee-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:16.521274 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:16.521096    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8c3306-98ff-4d83-a576-78aba00c24ee-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:17.034251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.034212    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:17.248643 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.248604    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:17.305922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.305845    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"
Oct 30 11:58:17.306420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.305842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k" event={"ID":"3e8c3306-98ff-4d83-a576-78aba00c24ee","Type":"ContainerDied","Data":"5bd9d590acb2349f36d888329e2a06c9e89518f62031ccce3534c7c2b57bf56e"}
Oct 30 11:58:17.306420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.305994    5060 scope.go:122] "RemoveContainer" containerID="a63a7dc87ffe3dbaef62234702bbbf27d0501c2665e679dba0b0842c9ec0f243"
Oct 30 11:58:17.393783 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.393747    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"]
Oct 30 11:58:17.406143 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:17.406105    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-9p68k"]
Oct 30 11:58:18.335357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:18.335320    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" path="/var/lib/kubelet/pods/3e8c3306-98ff-4d83-a576-78aba00c24ee/volumes"
Oct 30 11:58:19.118937 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:19.118896    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:19.391353 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:19.391257    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:19.638954 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:19.638907    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:19.875359 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:19.875305    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:58:20.062405 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.062368    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:20.229445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.229415    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 11:58:20.238977 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.238948    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"]
Oct 30 11:58:20.252960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.252926    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-caa7a0-6wf5x"]
Oct 30 11:58:20.336142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.336108    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac051f5-ff05-4819-aa40-cc7515a45ff3" path="/var/lib/kubelet/pods/4ac051f5-ff05-4819-aa40-cc7515a45ff3/volumes"
Oct 30 11:58:20.482370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.482268    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:20.768097 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:20.768004    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:22.280769 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:22.280724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:23.227919 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:23.227884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:23.408631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:23.408576    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:23.547871 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:23.547776    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"]
Oct 30 11:58:23.548629 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:23.548579    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy" containerID="cri-o://21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303" gracePeriod=30
Oct 30 11:58:23.604600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:23.604553    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:24.120172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:24.120136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:24.730114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:24.730076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:24.834758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:24.834719    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"]
Oct 30 11:58:24.846890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:24.846858    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73dffe-46kj2"]
Oct 30 11:58:24.879851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:24.879800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:25.024894 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:25.024794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:25.675550 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:25.675515    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"]
Oct 30 11:58:25.873894 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:25.873852    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:26.111036 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:26.110994    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"]
Oct 30 11:58:26.111506 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:26.111476    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy" containerID="cri-o://5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a" gracePeriod=30
Oct 30 11:58:26.335215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:26.335182    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98404b44-dddd-4671-8443-febd85319963" path="/var/lib/kubelet/pods/98404b44-dddd-4671-8443-febd85319963/volumes"
Oct 30 11:58:26.946756 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:26.946709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:27.124023 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:27.123985    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:27.274061 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:27.273977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:28.829557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.829474    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:58:28.916907 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.916863    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.916917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.916982    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917087 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917079    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917117    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917211 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917281 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917214    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917361 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917347    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.917424 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917375    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket\") pod \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\" (UID: \"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2\") "
Oct 30 11:58:28.918146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917803    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:28.918146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917946    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:28.918146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.917991    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:28.918146 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.918114    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:28.918371 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.918157    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:28.920141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.920102    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "kube-api-access-bkfvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:28.920540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.920506    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:28.920633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.920537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:28.923157 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.923124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token" pod "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" (UID: "d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:28.966218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:28.966178    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:29.019103 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019067    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019103 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019101    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019111    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfvr\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-kube-api-access-bkfvr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019122    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019130    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019138    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019146    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019155    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.019351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.019162    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:29.424035 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.424002    5060 generic.go:356] "Generic (PLEG): container finished" podID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerID="21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303" exitCode=0
Oct 30 11:58:29.424228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.424051    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"
Oct 30 11:58:29.424228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.424064    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" event={"ID":"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2","Type":"ContainerDied","Data":"21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303"}
Oct 30 11:58:29.424228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.424108    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w" event={"ID":"d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2","Type":"ContainerDied","Data":"63e6146e58608187005ae3f9978947128b6d3a2d801bc3303d61c7a74c3e06ae"}
Oct 30 11:58:29.424228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.424130    5060 scope.go:122] "RemoveContainer" containerID="21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303"
Oct 30 11:58:29.436204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.436184    5060 scope.go:122] "RemoveContainer" containerID="e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"
Oct 30 11:58:29.446794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.446771    5060 scope.go:122] "RemoveContainer" containerID="21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303"
Oct 30 11:58:29.447138 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:29.447116    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303\": container with ID starting with 21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303 not found: ID does not exist" containerID="21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303"
Oct 30 11:58:29.447184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.447148    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303"} err="failed to get container status \"21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303\": rpc error: code = NotFound desc = could not find container \"21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303\": container with ID starting with 21bd2c6c3de3116c5eae8f2bcee739de77e900a453ecb2f2fbb6b6a1c39f2303 not found: ID does not exist"
Oct 30 11:58:29.447184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.447165    5060 scope.go:122] "RemoveContainer" containerID="e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"
Oct 30 11:58:29.447432 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:29.447408    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd\": container with ID starting with e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd not found: ID does not exist" containerID="e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"
Oct 30 11:58:29.447616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.447442    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd"} err="failed to get container status \"e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd\": rpc error: code = NotFound desc = could not find container \"e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd\": container with ID starting with e3c43c1c1d5fb36f17e08f3bac5e5588369b0c7092b682164e114c85645ec6bd not found: ID does not exist"
Oct 30 11:58:29.478868 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.478831    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"]
Oct 30 11:58:29.491093 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:29.491057    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-648839-openshift-default-66c78757c-sg68w"]
Oct 30 11:58:30.335490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:30.335437    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" path="/var/lib/kubelet/pods/d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2/volumes"
Oct 30 11:58:30.851226 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:30.850416    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:31.403370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.403343    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:58:31.443276 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.443230    5060 generic.go:356] "Generic (PLEG): container finished" podID="672e173a-4991-4411-a7cf-306fbad68ccd" containerID="5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a" exitCode=0
Oct 30 11:58:31.443477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.443281    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"
Oct 30 11:58:31.443477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.443322    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" event={"ID":"672e173a-4991-4411-a7cf-306fbad68ccd","Type":"ContainerDied","Data":"5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a"}
Oct 30 11:58:31.443477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.443369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz" event={"ID":"672e173a-4991-4411-a7cf-306fbad68ccd","Type":"ContainerDied","Data":"a767685976b4f719e7fc7eaf1f5a13e8529d66d8c09d8a757cba034f6a42e70e"}
Oct 30 11:58:31.443477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.443394    5060 scope.go:122] "RemoveContainer" containerID="5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a"
Oct 30 11:58:31.450732 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.450705    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.450867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.450764    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.450867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.450814    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.450867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.450843    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.451070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.450873    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.451070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.451176 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451142    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:31.451221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:31.451273 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451249    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:31.451343 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.451399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451348    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.451485 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451391    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:31.451578 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451512    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:31.451781 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.451553    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy\") pod \"672e173a-4991-4411-a7cf-306fbad68ccd\" (UID: \"672e173a-4991-4411-a7cf-306fbad68ccd\") "
Oct 30 11:58:31.452182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.452159    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.452288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.452187    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672e173a-4991-4411-a7cf-306fbad68ccd-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.452288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.452201    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.452288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.452215    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.452288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.452228    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.453245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.453216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:31.453355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.453303    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "kube-api-access-czzwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:31.453681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.453665    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:31.453774 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.453761    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token" pod "672e173a-4991-4411-a7cf-306fbad68ccd" (UID: "672e173a-4991-4411-a7cf-306fbad68ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:31.455396 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.455376    5060 scope.go:122] "RemoveContainer" containerID="5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a"
Oct 30 11:58:31.455735 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:31.455714    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a\": container with ID starting with 5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a not found: ID does not exist" containerID="5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a"
Oct 30 11:58:31.455811 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.455745    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a"} err="failed to get container status \"5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a\": rpc error: code = NotFound desc = could not find container \"5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a\": container with ID starting with 5f9091e24086bd7712ae8ad95f2b19fabf03bb967a80f71ec6cf4ef58655314a not found: ID does not exist"
Oct 30 11:58:31.553851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.553811    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.553851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.553844    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672e173a-4991-4411-a7cf-306fbad68ccd-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.553851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.553852    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672e173a-4991-4411-a7cf-306fbad68ccd-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.553851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.553861    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzwd\" (UniqueName: \"kubernetes.io/projected/672e173a-4991-4411-a7cf-306fbad68ccd-kube-api-access-czzwd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:31.792015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.791981    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"]
Oct 30 11:58:31.804118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:31.804086    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-x4wlz"]
Oct 30 11:58:32.334968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:32.334934    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" path="/var/lib/kubelet/pods/672e173a-4991-4411-a7cf-306fbad68ccd/volumes"
Oct 30 11:58:33.210945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:33.210903    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:35.010983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:35.010941    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:35.374174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:35.374082    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:36.453812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.453775    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:36.885109 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.885019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:36.920287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.920251    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"]
Oct 30 11:58:36.921339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921309    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy"
Oct 30 11:58:36.921339 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921337    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921354    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921361    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921394    5060 cpu_manager.go:414] "containerMap: removing container" podUID="94aefcd2-20db-4f74-8287-6e40074b270f" containerName="payload-pre-processing"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921401    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94aefcd2-20db-4f74-8287-6e40074b270f" containerName="payload-pre-processing"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921423    5060 cpu_manager.go:414] "containerMap: removing container" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921431    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921477    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy"
Oct 30 11:58:36.921577 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921485    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy"
Oct 30 11:58:36.921998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921630    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8c3306-98ff-4d83-a576-78aba00c24ee" containerName="istio-proxy"
Oct 30 11:58:36.921998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921648    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy"
Oct 30 11:58:36.921998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921657    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d4ce99-923c-4bdd-b10b-b8a3ab2601e2" containerName="istio-proxy"
Oct 30 11:58:36.921998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921671    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="94aefcd2-20db-4f74-8287-6e40074b270f" containerName="payload-pre-processing"
Oct 30 11:58:36.921998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.921683    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="672e173a-4991-4411-a7cf-306fbad68ccd" containerName="istio-proxy"
Oct 30 11:58:36.926792 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.926767    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:36.950751 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.950718    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"]
Oct 30 11:58:36.959086 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:36.959056    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e25f4a-openshift-default-dockercfg-fp4tl\""
Oct 30 11:58:37.027519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.026004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.027735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.027557    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.027735 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.027688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.027903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.027880    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.028139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.028114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.028269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.028180    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5k5\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.028346 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.028304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.028412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.028341    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.028412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.028401    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.136204    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.136263    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.136319    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.136399    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.136438    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.137186    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.137311    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.137414    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.137478    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.139527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.137523    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5k5\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.140544 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.140517    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.140938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.140918    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.161478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.143688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.161478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.144222    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.161478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.144335    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.161478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.144490    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.212531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.209719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5k5\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.212531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.210087    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.250477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.250375    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:37.256886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.256853    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:37.439503 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.439390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:37.584282 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:37.584248    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb80fc4_9c8d_43d5_9ca7_27b07da634a5.slice/crio-111b68c615fe5a07e7cacb2d3d961c9527cf10e0cbd802ca6dbd5d55f4fe6bbf WatchSource:0}: Error finding container 111b68c615fe5a07e7cacb2d3d961c9527cf10e0cbd802ca6dbd5d55f4fe6bbf: Status 404 returned error can't find the container with id 111b68c615fe5a07e7cacb2d3d961c9527cf10e0cbd802ca6dbd5d55f4fe6bbf
Oct 30 11:58:37.585047 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.585018    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"]
Oct 30 11:58:37.587182 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.587140    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:37.587304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.587233    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:37.587304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:37.587275    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:38.521619 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:38.521582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" event={"ID":"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5","Type":"ContainerStarted","Data":"5aef57c1ff5eac80ca5883b31cfeca72c410655664574ae1f7a656d5186cd277"}
Oct 30 11:58:38.521858 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:38.521639    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" event={"ID":"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5","Type":"ContainerStarted","Data":"111b68c615fe5a07e7cacb2d3d961c9527cf10e0cbd802ca6dbd5d55f4fe6bbf"}
Oct 30 11:58:38.583272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:38.583210    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podStartSLOduration=2.583195273 podStartE2EDuration="2.583195273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:38.579477635 +0000 UTC m=+1722.374362120" watchObservedRunningTime="2026-10-30 11:58:38.583195273 +0000 UTC m=+1722.378079766"
Oct 30 11:58:38.904330 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:38.902604    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:39.251627 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:39.251592    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:39.257117 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:39.257089    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:39.529873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:39.529781    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:39.530851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:39.530833    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:58:40.697931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:40.697894    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn" podUID="9c717b3f-0861-454a-9789-a45c7aed058c" containerName="main" containerID="cri-o://4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3" gracePeriod=45
Oct 30 11:58:40.755297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:40.755263    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:41.072170 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.072144    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:58:41.197846 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.197811    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home\") pod \"9c717b3f-0861-454a-9789-a45c7aed058c\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") "
Oct 30 11:58:41.198217 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.198198    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm\") pod \"9c717b3f-0861-454a-9789-a45c7aed058c\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") "
Oct 30 11:58:41.198390 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.198317    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home" pod "9c717b3f-0861-454a-9789-a45c7aed058c" (UID: "9c717b3f-0861-454a-9789-a45c7aed058c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:41.198622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.198393    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache\") pod \"9c717b3f-0861-454a-9789-a45c7aed058c\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") "
Oct 30 11:58:41.198794 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.198772    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs\") pod \"9c717b3f-0861-454a-9789-a45c7aed058c\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") "
Oct 30 11:58:41.199304 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.199281    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache" pod "9c717b3f-0861-454a-9789-a45c7aed058c" (UID: "9c717b3f-0861-454a-9789-a45c7aed058c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:41.199529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.199512    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn\" (UniqueName: \"kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn\") pod \"9c717b3f-0861-454a-9789-a45c7aed058c\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") "
Oct 30 11:58:41.200444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.200399    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir" pod "9c717b3f-0861-454a-9789-a45c7aed058c" (UID: "9c717b3f-0861-454a-9789-a45c7aed058c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:41.200604 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.200579    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir\") pod \"9c717b3f-0861-454a-9789-a45c7aed058c\" (UID: \"9c717b3f-0861-454a-9789-a45c7aed058c\") "
Oct 30 11:58:41.201375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.201357    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:41.201555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.201541    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:41.201677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.201666    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:41.202136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.202111    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs" pod "9c717b3f-0861-454a-9789-a45c7aed058c" (UID: "9c717b3f-0861-454a-9789-a45c7aed058c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:41.202547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.202523    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm" pod "9c717b3f-0861-454a-9789-a45c7aed058c" (UID: "9c717b3f-0861-454a-9789-a45c7aed058c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:41.221834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.210915    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn" pod "9c717b3f-0861-454a-9789-a45c7aed058c" (UID: "9c717b3f-0861-454a-9789-a45c7aed058c"). InnerVolumeSpecName "kube-api-access-r2jsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:41.304538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.303530    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c717b3f-0861-454a-9789-a45c7aed058c-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:41.304538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.303569    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c717b3f-0861-454a-9789-a45c7aed058c-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:41.304538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.303584    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jsn\" (UniqueName: \"kubernetes.io/projected/9c717b3f-0861-454a-9789-a45c7aed058c-kube-api-access-r2jsn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:41.546491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.546442    5060 generic.go:356] "Generic (PLEG): container finished" podID="9c717b3f-0861-454a-9789-a45c7aed058c" containerID="4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3" exitCode=0
Oct 30 11:58:41.546712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.546480    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn" event={"ID":"9c717b3f-0861-454a-9789-a45c7aed058c","Type":"ContainerDied","Data":"4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3"}
Oct 30 11:58:41.546712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.546504    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"
Oct 30 11:58:41.546712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.546538    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn" event={"ID":"9c717b3f-0861-454a-9789-a45c7aed058c","Type":"ContainerDied","Data":"dcb2047c53e098f0032e99f61f712da5d2636a919401a69a5fba8ad457da9734"}
Oct 30 11:58:41.546712 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.546561    5060 scope.go:122] "RemoveContainer" containerID="4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3"
Oct 30 11:58:41.558075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.558045    5060 scope.go:122] "RemoveContainer" containerID="4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3"
Oct 30 11:58:41.558409 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:41.558386    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3\": container with ID starting with 4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3 not found: ID does not exist" containerID="4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3"
Oct 30 11:58:41.558502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.558420    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3"} err="failed to get container status \"4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3\": rpc error: code = NotFound desc = could not find container \"4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3\": container with ID starting with 4259d51a9a20183d0c65f71c57c956a5150585f17bcad74bd555426051dab8a3 not found: ID does not exist"
Oct 30 11:58:41.630008 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.629968    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"]
Oct 30 11:58:41.638481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:41.634321    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-caa7a0/ipp-route-caa7a0-kserve-8689d5d9bf-nzlgn"]
Oct 30 11:58:42.334532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.334493    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c717b3f-0861-454a-9789-a45c7aed058c" path="/var/lib/kubelet/pods/9c717b3f-0861-454a-9789-a45c7aed058c/volumes"
Oct 30 11:58:42.722660 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.722627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:42.866062 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.866026    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:42.965473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.965416    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"]
Oct 30 11:58:42.965668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.965638    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" containerID="cri-o://082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573" gracePeriod=30
Oct 30 11:58:42.983600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.983520    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"]
Oct 30 11:58:42.983808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.983745    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy" containerID="cri-o://c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5" gracePeriod=30
Oct 30 11:58:42.984640 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.984613    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"]
Oct 30 11:58:42.984847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.984823    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy" containerID="cri-o://89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f" gracePeriod=30
Oct 30 11:58:42.996001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.995971    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"]
Oct 30 11:58:42.996248 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:42.996219    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy" containerID="cri-o://3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b" gracePeriod=30
Oct 30 11:58:43.057509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:43.056321    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"]
Oct 30 11:58:43.057509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:43.056705    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" containerID="cri-o://87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052" gracePeriod=30
Oct 30 11:58:43.134319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:43.134276    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:43.284437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:43.284343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:44.121328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.121288    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:44.532851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.532809    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"]
Oct 30 11:58:44.533645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.533625    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9c717b3f-0861-454a-9789-a45c7aed058c" containerName="main"
Oct 30 11:58:44.533645 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.533645    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c717b3f-0861-454a-9789-a45c7aed058c" containerName="main"
Oct 30 11:58:44.533831 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.533794    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c717b3f-0861-454a-9789-a45c7aed058c" containerName="main"
Oct 30 11:58:44.539294 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.539267    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.560962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.560929    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-48e1f1-openshift-default-dockercfg-sbp2p\""
Oct 30 11:58:44.589309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.589272    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"]
Oct 30 11:58:44.629075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.629027    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:44.659957 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.659916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660024    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660094    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660174 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660179    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660225    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660257    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660305    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.660356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.660331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nl6\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764226    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764303    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764324    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764366    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764395    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764437    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.764477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764476    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nl6\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.765029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.765029 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.764654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.765141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.765063    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.776538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.765273    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.776538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.765839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.776538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.766040    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.776538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.766572    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.783188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.783117    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.798319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.797246    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.805229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.804560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.805229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.804793    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:44.808723 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.808696    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nl6\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6\") pod \"e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") " pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.856958 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.856923    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:44.862398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.862363    5060 patch_prober.go:29] interesting pod/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:58:44.862556 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:44.862428    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:58:45.237747 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:45.237714    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b9dfb1_f962_4e38_8c48_989fb1478455.slice/crio-259e593eea7087a4f0f79b082850b65e1903c27d16db782d649a52d5fd522c59 WatchSource:0}: Error finding container 259e593eea7087a4f0f79b082850b65e1903c27d16db782d649a52d5fd522c59: Status 404 returned error can't find the container with id 259e593eea7087a4f0f79b082850b65e1903c27d16db782d649a52d5fd522c59
Oct 30 11:58:45.239255 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.239226    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"]
Oct 30 11:58:45.240620 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.240574    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:45.240689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.240670    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:45.240726 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.240713    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:45.352138 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.352101    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:45.530531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.530178    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:45.607480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.606611    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" event={"ID":"d6b9dfb1-f962-4e38-8c48-989fb1478455","Type":"ContainerStarted","Data":"e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"}
Oct 30 11:58:45.607480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.606674    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" event={"ID":"d6b9dfb1-f962-4e38-8c48-989fb1478455","Type":"ContainerStarted","Data":"259e593eea7087a4f0f79b082850b65e1903c27d16db782d649a52d5fd522c59"}
Oct 30 11:58:45.656624 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.656556    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podStartSLOduration=1.656540697 podStartE2EDuration="1.656540697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:45.655636267 +0000 UTC m=+1729.450520763" watchObservedRunningTime="2026-10-30 11:58:45.656540697 +0000 UTC m=+1729.451425193"
Oct 30 11:58:45.857567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:45.857001    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:46.580545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:46.580503    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b79r6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:58:46.580921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:46.580571    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:58:46.863998 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:46.863910    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:47.623358 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:47.623326    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:47.624586 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:47.624564    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:58:47.654678 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:47.654646    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:48.348077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.346142    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:48.350420 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.350398    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:58:48.392755 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.392727    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:58:48.398070 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.397731    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:58:48.403710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.403682    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:58:48.447468 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.447433    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:58:48.523075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.522981    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.523075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.523319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523091    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.523319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523125    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.523319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523163    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.523319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523197    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.523319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.523319 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523289    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523328    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523373    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523401    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523431    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523484    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523533    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523565    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523598    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523660    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523688    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523733    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523783    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.523873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523873    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.524658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.524658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523961    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.524658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.523995    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.524658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.524027    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.524658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.524066    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.524658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.524405    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.524972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.524662    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.525489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525229    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.525489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:48.525750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.524095    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.525750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525632    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs\") pod \"aff70f28-f944-41c7-9aa4-ae38ecab3790\" (UID: \"aff70f28-f944-41c7-9aa4-ae38ecab3790\") "
Oct 30 11:58:48.525750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525697    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.525750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525729    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.525967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.525967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525839    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.525967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525860    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs\") pod \"53c7d92f-8eef-4be9-b3d7-e0e877130074\" (UID: \"53c7d92f-8eef-4be9-b3d7-e0e877130074\") "
Oct 30 11:58:48.525967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525899    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket\") pod \"97c1fa20-c3f7-4606-9dae-8b81572ef711\" (UID: \"97c1fa20-c3f7-4606-9dae-8b81572ef711\") "
Oct 30 11:58:48.525967 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.525936    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket\") pod \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\" (UID: \"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f\") "
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.526318    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.526534    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.526744    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.526834    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527029    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527068    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.526792    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527132    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527193    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527209    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527224    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527239    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527441    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527649    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.527821    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.528262 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.528257    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.529391 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.529364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.529718 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.529519    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:48.529953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.529840    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:48.529953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.529850    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "kube-api-access-hgjc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.530704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.530676    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:48.531035 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.530986    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.534596    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "kube-api-access-g8sc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.534780    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.534655    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.534884    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.534920    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "kube-api-access-5j5hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.534964    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.535028    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "kube-api-access-ptdnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.535063    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.535135    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.535183    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.535494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.535194    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:48.536398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.536371    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy" pod "97c1fa20-c3f7-4606-9dae-8b81572ef711" (UID: "97c1fa20-c3f7-4606-9dae-8b81572ef711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.537593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.536671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo" pod "aff70f28-f944-41c7-9aa4-ae38ecab3790" (UID: "aff70f28-f944-41c7-9aa4-ae38ecab3790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:48.537849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.537771    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy" pod "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" (UID: "e69ee38c-c8c8-4c63-8ce1-5d03b443c54f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.539504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.538544    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy" pod "53c7d92f-8eef-4be9-b3d7-e0e877130074" (UID: "53c7d92f-8eef-4be9-b3d7-e0e877130074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.629875    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.629914    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.629957    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630040    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630256    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630399    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630539    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630570    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert\") pod \"96900fe1-8b8d-4c32-895b-3fefda261c4e\" (UID: \"96900fe1-8b8d-4c32-895b-3fefda261c4e\") "
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.630999    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff70f28-f944-41c7-9aa4-ae38ecab3790-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631022    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdnn\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-kube-api-access-ptdnn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631033    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631042    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j5hh\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-kube-api-access-5j5hh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631050    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631059    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631072    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631085    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631096    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631104    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631112    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631121    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631130    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjc4\" (UniqueName: \"kubernetes.io/projected/53c7d92f-8eef-4be9-b3d7-e0e877130074-kube-api-access-hgjc4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631138    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631149    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7d92f-8eef-4be9-b3d7-e0e877130074-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631163    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631175    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631184    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631195    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631218    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c1fa20-c3f7-4606-9dae-8b81572ef711-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631231    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff70f28-f944-41c7-9aa4-ae38ecab3790-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631244    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631257    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631269    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631281    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8sc9\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-kube-api-access-g8sc9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631289    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff70f28-f944-41c7-9aa4-ae38ecab3790-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631297    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7d92f-8eef-4be9-b3d7-e0e877130074-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631305    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631313    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.631589 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631325    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c1fa20-c3f7-4606-9dae-8b81572ef711-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.633911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631679    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:48.633911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.631940    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.633911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.632093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.633911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.632244    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.633911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.633680    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.635014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.634985    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "kube-api-access-whpw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.636340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.636309    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:48.636340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.636328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:58:48.636849 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.636829    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo" pod "96900fe1-8b8d-4c32-895b-3fefda261c4e" (UID: "96900fe1-8b8d-4c32-895b-3fefda261c4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:58:48.643377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.642988    5060 generic.go:356] "Generic (PLEG): container finished" podID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerID="c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5" exitCode=0
Oct 30 11:58:48.643377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.643051    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" event={"ID":"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f","Type":"ContainerDied","Data":"c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5"}
Oct 30 11:58:48.643377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.643068    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"
Oct 30 11:58:48.643377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.643194    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd" event={"ID":"e69ee38c-c8c8-4c63-8ce1-5d03b443c54f","Type":"ContainerDied","Data":"6c7446d575d3cce449eed9ce49a74feea302d455ed45b7d23acd87eacfc14066"}
Oct 30 11:58:48.643377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.643232    5060 scope.go:122] "RemoveContainer" containerID="c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5"
Oct 30 11:58:48.645568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.645542    5060 generic.go:356] "Generic (PLEG): container finished" podID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerID="89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f" exitCode=0
Oct 30 11:58:48.645689 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.645671    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"
Oct 30 11:58:48.645753 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.645668    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" event={"ID":"97c1fa20-c3f7-4606-9dae-8b81572ef711","Type":"ContainerDied","Data":"89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f"}
Oct 30 11:58:48.645847 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.645827    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d" event={"ID":"97c1fa20-c3f7-4606-9dae-8b81572ef711","Type":"ContainerDied","Data":"b6011da0b4b6a8860f2da5f8f66959a28142eb5bb9d003bf37fb20a919885da6"}
Oct 30 11:58:48.650721 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.650644    5060 generic.go:356] "Generic (PLEG): container finished" podID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerID="082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573" exitCode=0
Oct 30 11:58:48.650867 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.650829    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"
Oct 30 11:58:48.651130 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.651105    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" event={"ID":"aff70f28-f944-41c7-9aa4-ae38ecab3790","Type":"ContainerDied","Data":"082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573"}
Oct 30 11:58:48.651220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.651159    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6" event={"ID":"aff70f28-f944-41c7-9aa4-ae38ecab3790","Type":"ContainerDied","Data":"f459932596b0cdafbf77940e1b39ec4116c36a1d3a4f0cc11b94a529f8445d80"}
Oct 30 11:58:48.654908 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.654878    5060 generic.go:356] "Generic (PLEG): container finished" podID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerID="87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052" exitCode=0
Oct 30 11:58:48.655038 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.655005    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" event={"ID":"53c7d92f-8eef-4be9-b3d7-e0e877130074","Type":"ContainerDied","Data":"87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052"}
Oct 30 11:58:48.655102 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.655052    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4" event={"ID":"53c7d92f-8eef-4be9-b3d7-e0e877130074","Type":"ContainerDied","Data":"6d64537a5f79e00895ac5fd5a118573b971b09741ac5ef7e1c58511639369227"}
Oct 30 11:58:48.655172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.655158    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"
Oct 30 11:58:48.660693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.660672    5060 scope.go:122] "RemoveContainer" containerID="c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5"
Oct 30 11:58:48.661408 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:48.661280    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5\": container with ID starting with c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5 not found: ID does not exist" containerID="c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5"
Oct 30 11:58:48.661633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.661321    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5"} err="failed to get container status \"c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5\": rpc error: code = NotFound desc = could not find container \"c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5\": container with ID starting with c225dc0c9a3b4239f62199dfcfd3c12e554e66551b9e13a5954e7138b82c70f5 not found: ID does not exist"
Oct 30 11:58:48.661633 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.661613    5060 scope.go:122] "RemoveContainer" containerID="89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f"
Oct 30 11:58:48.664504 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.664444    5060 generic.go:356] "Generic (PLEG): container finished" podID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerID="3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b" exitCode=0
Oct 30 11:58:48.664945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.664924    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"
Oct 30 11:58:48.665052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.664964    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" event={"ID":"96900fe1-8b8d-4c32-895b-3fefda261c4e","Type":"ContainerDied","Data":"3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b"}
Oct 30 11:58:48.665052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.665022    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5" event={"ID":"96900fe1-8b8d-4c32-895b-3fefda261c4e","Type":"ContainerDied","Data":"99ee79b8cccd1cd9f3eacbd020144379f523c2d6b5171744f3c7e31b04e57f5d"}
Oct 30 11:58:48.677615 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.677595    5060 scope.go:122] "RemoveContainer" containerID="89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f"
Oct 30 11:58:48.678293 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:48.678262    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f\": container with ID starting with 89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f not found: ID does not exist" containerID="89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f"
Oct 30 11:58:48.678377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.678304    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f"} err="failed to get container status \"89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f\": rpc error: code = NotFound desc = could not find container \"89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f\": container with ID starting with 89cddd958235289d5920916b9077b89713f67872017f6b9a2b31b1f70e19822f not found: ID does not exist"
Oct 30 11:58:48.678377 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.678329    5060 scope.go:122] "RemoveContainer" containerID="082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573"
Oct 30 11:58:48.689206 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.689182    5060 scope.go:122] "RemoveContainer" containerID="082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573"
Oct 30 11:58:48.689591 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:48.689555    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573\": container with ID starting with 082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573 not found: ID does not exist" containerID="082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573"
Oct 30 11:58:48.689659 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.689597    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573"} err="failed to get container status \"082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573\": rpc error: code = NotFound desc = could not find container \"082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573\": container with ID starting with 082371a2f3fb52596553e3be33b0500f940edf475d9001c3004eb3753faab573 not found: ID does not exist"
Oct 30 11:58:48.689659 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.689624    5060 scope.go:122] "RemoveContainer" containerID="87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052"
Oct 30 11:58:48.702888 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.702868    5060 scope.go:122] "RemoveContainer" containerID="bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"
Oct 30 11:58:48.713881 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.713856    5060 scope.go:122] "RemoveContainer" containerID="87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052"
Oct 30 11:58:48.714266 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:48.714241    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052\": container with ID starting with 87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052 not found: ID does not exist" containerID="87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052"
Oct 30 11:58:48.714356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.714279    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052"} err="failed to get container status \"87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052\": rpc error: code = NotFound desc = could not find container \"87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052\": container with ID starting with 87634a2bce5909a489f353cc6d74f54eda9c856d7193369aa6e5365c7e807052 not found: ID does not exist"
Oct 30 11:58:48.714356 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.714306    5060 scope.go:122] "RemoveContainer" containerID="bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"
Oct 30 11:58:48.714601 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:48.714577    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993\": container with ID starting with bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993 not found: ID does not exist" containerID="bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"
Oct 30 11:58:48.714677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.714609    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993"} err="failed to get container status \"bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993\": rpc error: code = NotFound desc = could not find container \"bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993\": container with ID starting with bfc0edb61cd52999842405eff50667417b1e19faca80a976ca3016877b792993 not found: ID does not exist"
Oct 30 11:58:48.714677 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.714629    5060 scope.go:122] "RemoveContainer" containerID="3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b"
Oct 30 11:58:48.725747 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.725726    5060 scope.go:122] "RemoveContainer" containerID="3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b"
Oct 30 11:58:48.726100 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:48.726080    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b\": container with ID starting with 3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b not found: ID does not exist" containerID="3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b"
Oct 30 11:58:48.726167 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.726109    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b"} err="failed to get container status \"3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b\": rpc error: code = NotFound desc = could not find container \"3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b\": container with ID starting with 3a240e2789e3e85f7509a24a199280c9d19bc3dc89611151307da218703a5d8b not found: ID does not exist"
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734346    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734379    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96900fe1-8b8d-4c32-895b-3fefda261c4e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734392    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734407    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734418    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734444    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpw4\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-kube-api-access-whpw4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734478    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734505    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.734545 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.734517    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96900fe1-8b8d-4c32-895b-3fefda261c4e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:48.749720 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.749684    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"]
Oct 30 11:58:48.756183 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.756150    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tcnb5"]
Oct 30 11:58:48.799862 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.799780    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"]
Oct 30 11:58:48.807075 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.807036    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73dffe-openshift-default-768c786b9c-ww8r4"]
Oct 30 11:58:48.845057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.845034    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"]
Oct 30 11:58:48.854001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.853971    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tzctd"]
Oct 30 11:58:48.898863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.898825    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"]
Oct 30 11:58:48.909606 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.909573    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b79r6"]
Oct 30 11:58:48.945204 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.945165    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"]
Oct 30 11:58:48.951074 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:48.951046    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z262d"]
Oct 30 11:58:49.107716 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.107622    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"]
Oct 30 11:58:49.108163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108146    5060 cpu_manager.go:414] "containerMap: removing container" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy"
Oct 30 11:58:49.108163 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108162    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108182    5060 cpu_manager.go:414] "containerMap: removing container" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108188    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108196    5060 cpu_manager.go:414] "containerMap: removing container" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108201    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108219    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108223    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108248    5060 cpu_manager.go:414] "containerMap: removing container" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy"
Oct 30 11:58:49.108300 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108254    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy"
Oct 30 11:58:49.108688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108332    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" containerName="istio-proxy"
Oct 30 11:58:49.108688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108343    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy"
Oct 30 11:58:49.108688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108349    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" containerName="istio-proxy"
Oct 30 11:58:49.108688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108356    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" containerName="istio-proxy"
Oct 30 11:58:49.108688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108362    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy"
Oct 30 11:58:49.108688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.108369    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" containerName="istio-proxy"
Oct 30 11:58:49.113418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.113395    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.128446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.128417    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:58:49.128666 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.128584    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:58:49.128666 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.128600    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:58:49.128666 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.128634    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:58:49.129881 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.129860    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"]
Oct 30 11:58:49.240977 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.240937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.241141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.240987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.241141 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.241016    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbrz\" (UniqueName: \"kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.341924 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.341883    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.342110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.341951    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.342110 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.341975    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbrz\" (UniqueName: \"kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.342738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.342712    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.344564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.344534    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.365921 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.365849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbrz\" (UniqueName: \"kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.429121 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.429081    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:49.743324 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:49.743294    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513a0739_2c70_4e69_8593_d242f99a77fc.slice/crio-261ef0d6ee19d800f1a1c4b8f1f588fc2b06f488dd9b02a73c9db872675163be WatchSource:0}: Error finding container 261ef0d6ee19d800f1a1c4b8f1f588fc2b06f488dd9b02a73c9db872675163be: Status 404 returned error can't find the container with id 261ef0d6ee19d800f1a1c4b8f1f588fc2b06f488dd9b02a73c9db872675163be
Oct 30 11:58:49.743701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:49.743675    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"]
Oct 30 11:58:49.858887 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:58:49.858852    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:58:50.024708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.024619    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:50.168015 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.167973    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:50.201331 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.201292    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"]
Oct 30 11:58:50.202131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.202111    5060 cpu_manager.go:414] "containerMap: removing container" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy"
Oct 30 11:58:50.202202 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.202136    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" containerName="istio-proxy"
Oct 30 11:58:50.206297 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.206273    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.224184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.222233    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"]
Oct 30 11:58:50.224184 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.222706    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-48e1f1-dockercfg-rdq88\""
Oct 30 11:58:50.227072 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.227039    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-48e1f1\""
Oct 30 11:58:50.252549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.252518    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.252549 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.252553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h7t\" (UniqueName: \"kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.348530 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.339735    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:50.355350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.354861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.355350 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.354923    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h7t\" (UniqueName: \"kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.357071 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.356365    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.367580 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.366295    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c7d92f-8eef-4be9-b3d7-e0e877130074" path="/var/lib/kubelet/pods/53c7d92f-8eef-4be9-b3d7-e0e877130074/volumes"
Oct 30 11:58:50.367580 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.367345    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96900fe1-8b8d-4c32-895b-3fefda261c4e" path="/var/lib/kubelet/pods/96900fe1-8b8d-4c32-895b-3fefda261c4e/volumes"
Oct 30 11:58:50.368145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.368122    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c1fa20-c3f7-4606-9dae-8b81572ef711" path="/var/lib/kubelet/pods/97c1fa20-c3f7-4606-9dae-8b81572ef711/volumes"
Oct 30 11:58:50.372642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.369254    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff70f28-f944-41c7-9aa4-ae38ecab3790" path="/var/lib/kubelet/pods/aff70f28-f944-41c7-9aa4-ae38ecab3790/volumes"
Oct 30 11:58:50.372642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.369950    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69ee38c-c8c8-4c63-8ce1-5d03b443c54f" path="/var/lib/kubelet/pods/e69ee38c-c8c8-4c63-8ce1-5d03b443c54f/volumes"
Oct 30 11:58:50.372642 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.370389    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"]
Oct 30 11:58:50.392481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.391238    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"]
Oct 30 11:58:50.392481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.391378    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.415063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.412267    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h7t\" (UniqueName: \"kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.531373 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.531323    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:50.558218 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.558182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.558359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.558247    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7wh\" (UniqueName: \"kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh\") pod \"payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.659286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.659190    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.659286 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.659235    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7wh\" (UniqueName: \"kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh\") pod \"payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.659980 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.659954    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.685063 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.685026    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7wh\" (UniqueName: \"kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh\") pod \"payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.690904 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.690866    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm" event={"ID":"513a0739-2c70-4e69-8593-d242f99a77fc","Type":"ContainerStarted","Data":"4c814fae7d9445e989dbfb0e22e95a2b75b350641f6f8cf9af4d0c22f574c138"}
Oct 30 11:58:50.691056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.690934    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm" event={"ID":"513a0739-2c70-4e69-8593-d242f99a77fc","Type":"ContainerStarted","Data":"261ef0d6ee19d800f1a1c4b8f1f588fc2b06f488dd9b02a73c9db872675163be"}
Oct 30 11:58:50.726582 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.726532    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:50.733123 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.733075    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm" podStartSLOduration=1.733060448 podStartE2EDuration="1.733060448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:50.730494511 +0000 UTC m=+1734.525379002" watchObservedRunningTime="2026-10-30 11:58:50.733060448 +0000 UTC m=+1734.527944941"
Oct 30 11:58:50.840575 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:50.839673    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eb3845_8e72_4f96_b978_8e0280920fdd.slice/crio-3b08d2e99598138a1531cd2a5494ec3d96db10356715e609171c367ce4de9a42 WatchSource:0}: Error finding container 3b08d2e99598138a1531cd2a5494ec3d96db10356715e609171c367ce4de9a42: Status 404 returned error can't find the container with id 3b08d2e99598138a1531cd2a5494ec3d96db10356715e609171c367ce4de9a42
Oct 30 11:58:50.840575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:50.840242    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"]
Oct 30 11:58:51.036440 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:51.036390    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda758c044_bc32_4683_8da5_b5351a394fe1.slice/crio-2f8e21974ef6b5edf5042783781361e2dc7c261147154b3714f3973af10cadd0 WatchSource:0}: Error finding container 2f8e21974ef6b5edf5042783781361e2dc7c261147154b3714f3973af10cadd0: Status 404 returned error can't find the container with id 2f8e21974ef6b5edf5042783781361e2dc7c261147154b3714f3973af10cadd0
Oct 30 11:58:51.037554 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.037529    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"]
Oct 30 11:58:51.682822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.682786    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:51.705332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.705288    5060 generic.go:356] "Generic (PLEG): container finished" podID="513a0739-2c70-4e69-8593-d242f99a77fc" containerID="4c814fae7d9445e989dbfb0e22e95a2b75b350641f6f8cf9af4d0c22f574c138" exitCode=0
Oct 30 11:58:51.705532 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.705369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm" event={"ID":"513a0739-2c70-4e69-8593-d242f99a77fc","Type":"ContainerDied","Data":"4c814fae7d9445e989dbfb0e22e95a2b75b350641f6f8cf9af4d0c22f574c138"}
Oct 30 11:58:51.707433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.707408    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb" event={"ID":"06eb3845-8e72-4f96-b978-8e0280920fdd","Type":"ContainerStarted","Data":"6c5bff1ab1f71d2133c28a634380333722bfa271d2dbb7f5024d3068ec4fe93a"}
Oct 30 11:58:51.707595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.707494    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb" event={"ID":"06eb3845-8e72-4f96-b978-8e0280920fdd","Type":"ContainerStarted","Data":"3b08d2e99598138a1531cd2a5494ec3d96db10356715e609171c367ce4de9a42"}
Oct 30 11:58:51.707595 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.707529    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:58:51.709414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.709391    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6" event={"ID":"a758c044-bc32-4683-8da5-b5351a394fe1","Type":"ContainerStarted","Data":"e0403a324eb6b645564d91dad85094660e268e9aa9e8facc60d22ee24afd3da1"}
Oct 30 11:58:51.709539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.709437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6" event={"ID":"a758c044-bc32-4683-8da5-b5351a394fe1","Type":"ContainerStarted","Data":"2f8e21974ef6b5edf5042783781361e2dc7c261147154b3714f3973af10cadd0"}
Oct 30 11:58:51.709585 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.709540    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:58:51.821565 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.821499    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6" podStartSLOduration=1.821479182 podStartE2EDuration="1.821479182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:51.818966942 +0000 UTC m=+1735.613851435" watchObservedRunningTime="2026-10-30 11:58:51.821479182 +0000 UTC m=+1735.616363678"
Oct 30 11:58:51.914037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.913984    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb" podStartSLOduration=1.913966592 podStartE2EDuration="1.913966592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:51.910508178 +0000 UTC m=+1735.705392669" watchObservedRunningTime="2026-10-30 11:58:51.913966592 +0000 UTC m=+1735.708851125"
Oct 30 11:58:51.998478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:51.997514    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:52.163612 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.163573    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:52.453538 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.453506    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:52.876515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.876470    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:52.887911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.887863    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz\" (UniqueName: \"kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz\") pod \"513a0739-2c70-4e69-8593-d242f99a77fc\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") "
Oct 30 11:58:52.888104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.887972    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca\") pod \"513a0739-2c70-4e69-8593-d242f99a77fc\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") "
Oct 30 11:58:52.888104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.888097    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca\") pod \"513a0739-2c70-4e69-8593-d242f99a77fc\" (UID: \"513a0739-2c70-4e69-8593-d242f99a77fc\") "
Oct 30 11:58:52.888487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.888440    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca" pod "513a0739-2c70-4e69-8593-d242f99a77fc" (UID: "513a0739-2c70-4e69-8593-d242f99a77fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:52.888798 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.888637    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:52.890246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.890224    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz" pod "513a0739-2c70-4e69-8593-d242f99a77fc" (UID: "513a0739-2c70-4e69-8593-d242f99a77fc"). InnerVolumeSpecName "kube-api-access-6gbrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:52.890548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.890530    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca" pod "513a0739-2c70-4e69-8593-d242f99a77fc" (UID: "513a0739-2c70-4e69-8593-d242f99a77fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:52.989959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.989876    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513a0739-2c70-4e69-8593-d242f99a77fc-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:52.989959 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:52.989904    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gbrz\" (UniqueName: \"kubernetes.io/projected/513a0739-2c70-4e69-8593-d242f99a77fc-kube-api-access-6gbrz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:58:53.162694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.162663    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:53.459151 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.459106    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:53.600479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.597987    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"]
Oct 30 11:58:53.600479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.598827    5060 cpu_manager.go:414] "containerMap: removing container" podUID="513a0739-2c70-4e69-8593-d242f99a77fc" containerName="revoke-keys"
Oct 30 11:58:53.600479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.598846    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513a0739-2c70-4e69-8593-d242f99a77fc" containerName="revoke-keys"
Oct 30 11:58:53.600479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.598987    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="513a0739-2c70-4e69-8593-d242f99a77fc" containerName="revoke-keys"
Oct 30 11:58:53.606498 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.603285    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"]
Oct 30 11:58:53.608953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.608000    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.608953 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.608776    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.649725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.633005    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b920f1-openshift-default-dockercfg-97q6p\""
Oct 30 11:58:53.649725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.633146    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e25f4a-dockercfg-c4tjv\""
Oct 30 11:58:53.649725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.633210    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e25f4a\""
Oct 30 11:58:53.649725 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.637982    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"]
Oct 30 11:58:53.663488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.653089    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"]
Oct 30 11:58:53.663488 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.654992    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:53.696982 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.696836    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"]
Oct 30 11:58:53.699142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699106    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699270    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699407 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699306    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699394    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699580    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjm9\" (UniqueName: \"kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9\") pod \"payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.699730 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699702    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzl4\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699730 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699741    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.699922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699781    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.699922 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.699798    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.701337 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.701319    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.720634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.720601    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"]
Oct 30 11:58:53.763771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.763728    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm" event={"ID":"513a0739-2c70-4e69-8593-d242f99a77fc","Type":"ContainerDied","Data":"261ef0d6ee19d800f1a1c4b8f1f588fc2b06f488dd9b02a73c9db872675163be"}
Oct 30 11:58:53.763946 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.763799    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="261ef0d6ee19d800f1a1c4b8f1f588fc2b06f488dd9b02a73c9db872675163be"
Oct 30 11:58:53.763946 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.763891    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.800794    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzl4\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.800846    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.800893    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.800921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.800958    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q85\" (UniqueName: \"kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85\") pod \"payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801015    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801045    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801089    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801121    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801145    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801182    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801205    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801257    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjm9\" (UniqueName: \"kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9\") pod \"payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.801994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.802345    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.802739    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.803034    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.803469    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.805834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.803705    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.811494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.808758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.811494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.808996    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.822938 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.822866    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:53.825895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.825864    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.826058 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.825870    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjm9\" (UniqueName: \"kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9\") pod \"payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:53.826830 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.826436    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzl4\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4\") pod \"e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") " pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.902264 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.902222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q85\" (UniqueName: \"kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85\") pod \"payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.902465 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.902326    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.903108 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.903085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.929285 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.929242    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q85\" (UniqueName: \"kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85\") pod \"payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:53.960041 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.960002    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:53.965180 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.965142    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:53.968032 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:53.967998    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:54.074631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.074024    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:54.318741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.315634    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"]
Oct 30 11:58:54.318940 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.318911    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"]
Oct 30 11:58:54.323405 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:54.321176    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55df9ef_cccb_41c5_a82f_841bd60973dd.slice/crio-487de1283ddf79737fe77a70b712f48a3f0e9bbc87d4cc3a157506b0cb862dcc WatchSource:0}: Error finding container 487de1283ddf79737fe77a70b712f48a3f0e9bbc87d4cc3a157506b0cb862dcc: Status 404 returned error can't find the container with id 487de1283ddf79737fe77a70b712f48a3f0e9bbc87d4cc3a157506b0cb862dcc
Oct 30 11:58:54.323405 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:54.322276    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0952119e_66dd_4e14_95f0_f5971be5c762.slice/crio-cf39234789136cc67b71b4865d90e5bd67829041271933806eb9e5805c05f576 WatchSource:0}: Error finding container cf39234789136cc67b71b4865d90e5bd67829041271933806eb9e5805c05f576: Status 404 returned error can't find the container with id cf39234789136cc67b71b4865d90e5bd67829041271933806eb9e5805c05f576
Oct 30 11:58:54.325567 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.325534    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:54.325701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.325618    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:54.325701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.325646    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:58:54.414348 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.414319    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"]
Oct 30 11:58:54.415435 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:58:54.415094    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4211a8d7_2b8a_48d4_b747_09beb30621b0.slice/crio-39338a39018d70e22ccc33df9c0cdb6e81a3c651d6fb44a45de8547ddf83ab39 WatchSource:0}: Error finding container 39338a39018d70e22ccc33df9c0cdb6e81a3c651d6fb44a45de8547ddf83ab39: Status 404 returned error can't find the container with id 39338a39018d70e22ccc33df9c0cdb6e81a3c651d6fb44a45de8547ddf83ab39
Oct 30 11:58:54.776662 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.776629    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" event={"ID":"0952119e-66dd-4e14-95f0-f5971be5c762","Type":"ContainerStarted","Data":"71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"}
Oct 30 11:58:54.776883 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.776702    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" event={"ID":"0952119e-66dd-4e14-95f0-f5971be5c762","Type":"ContainerStarted","Data":"cf39234789136cc67b71b4865d90e5bd67829041271933806eb9e5805c05f576"}
Oct 30 11:58:54.779324 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.779283    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" event={"ID":"4211a8d7-2b8a-48d4-b747-09beb30621b0","Type":"ContainerStarted","Data":"34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d"}
Oct 30 11:58:54.779505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.779344    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" event={"ID":"4211a8d7-2b8a-48d4-b747-09beb30621b0","Type":"ContainerStarted","Data":"39338a39018d70e22ccc33df9c0cdb6e81a3c651d6fb44a45de8547ddf83ab39"}
Oct 30 11:58:54.779505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.779379    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:58:54.781509 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.781428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" event={"ID":"e55df9ef-cccb-41c5-a82f-841bd60973dd","Type":"ContainerStarted","Data":"0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c"}
Oct 30 11:58:54.781638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.781514    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:58:54.781638 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.781545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" event={"ID":"e55df9ef-cccb-41c5-a82f-841bd60973dd","Type":"ContainerStarted","Data":"487de1283ddf79737fe77a70b712f48a3f0e9bbc87d4cc3a157506b0cb862dcc"}
Oct 30 11:58:54.823094 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.823004    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podStartSLOduration=1.8229842029999999 podStartE2EDuration="1.822984203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:54.821813442 +0000 UTC m=+1738.616697954" watchObservedRunningTime="2026-10-30 11:58:54.822984203 +0000 UTC m=+1738.617868695"
Oct 30 11:58:54.867238 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.867169    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" podStartSLOduration=1.867147398 podStartE2EDuration="1.867147398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:54.86268621 +0000 UTC m=+1738.657570712" watchObservedRunningTime="2026-10-30 11:58:54.867147398 +0000 UTC m=+1738.662031901"
Oct 30 11:58:54.906859 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.906791    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" podStartSLOduration=1.906773046 podStartE2EDuration="1.906773046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:54.902749186 +0000 UTC m=+1738.697633679" watchObservedRunningTime="2026-10-30 11:58:54.906773046 +0000 UTC m=+1738.701657541"
Oct 30 11:58:54.960552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:54.960490    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:55.461283 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:55.461247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:55.966345 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:55.966315    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:55.996949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:55.996898    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:56.150761 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:56.150723    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:56.799251 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:56.799210    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:56.800359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:56.800339    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 11:58:57.353332 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:57.353306    5060 scope.go:122] "RemoveContainer" containerID="b63378b3b21f79000c5e2976179af55c1e5633013b9a356349bd0926519cabd6"
Oct 30 11:58:57.589918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:57.589874    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:57.765661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:57.765624    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:59.317902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.317864    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:59.456575 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.456534    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"]
Oct 30 11:58:59.462508 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.462485    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.476162 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.476136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:59.478685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.478663    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b920f1\""
Oct 30 11:58:59.480744 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.480721    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"]
Oct 30 11:58:59.489557 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.489526    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b920f1-dockercfg-5sbvm\""
Oct 30 11:58:59.577333 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.577236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6gr\" (UniqueName: \"kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr\") pod \"payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.577333 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.577288    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.638766 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.636129    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"]
Oct 30 11:58:59.641357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.641320    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"]
Oct 30 11:58:59.641523 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.641438    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.678150 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.678113    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44dl\" (UniqueName: \"kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl\") pod \"payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.678355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.678182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.678355 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.678338    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6gr\" (UniqueName: \"kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr\") pod \"payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.678439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.678378    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.679635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.679606    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.702260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.702226    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6gr\" (UniqueName: \"kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr\") pod \"payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.756437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.756401    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"]
Oct 30 11:58:59.775800 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.775769    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:58:59.779413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.779102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44dl\" (UniqueName: \"kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl\") pod \"payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.779413 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.779152    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.780077 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.780050    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.789851 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.789827    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"]
Oct 30 11:58:59.805597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.805566    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44dl\" (UniqueName: \"kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl\") pod \"payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:58:59.934668 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.934627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:58:59.956119 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:58:59.956085    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:59:00.085307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.085265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:00.093592 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:00.093558    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bb5375_fe10_4dca_8526_d82eef3fb816.slice/crio-2ac31e64a0db840c8bb4dad32d2fe1cf0dcb07152cbe38aa4736bbc781ffe551 WatchSource:0}: Error finding container 2ac31e64a0db840c8bb4dad32d2fe1cf0dcb07152cbe38aa4736bbc781ffe551: Status 404 returned error can't find the container with id 2ac31e64a0db840c8bb4dad32d2fe1cf0dcb07152cbe38aa4736bbc781ffe551
Oct 30 11:59:00.094359 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.094335    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"]
Oct 30 11:59:00.267748 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:00.267719    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc6036_09f8_4b92_a48d_706884607ba8.slice/crio-ae9da01000f27ec19c4dfcbd931fddb021b203fe57da511e39546ede85959561 WatchSource:0}: Error finding container ae9da01000f27ec19c4dfcbd931fddb021b203fe57da511e39546ede85959561: Status 404 returned error can't find the container with id ae9da01000f27ec19c4dfcbd931fddb021b203fe57da511e39546ede85959561
Oct 30 11:59:00.268780 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.268749    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"]
Oct 30 11:59:00.833421 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.833384    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk" event={"ID":"31bb5375-fe10-4dca-8526-d82eef3fb816","Type":"ContainerStarted","Data":"392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c"}
Oct 30 11:59:00.833942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.833523    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk" event={"ID":"31bb5375-fe10-4dca-8526-d82eef3fb816","Type":"ContainerStarted","Data":"2ac31e64a0db840c8bb4dad32d2fe1cf0dcb07152cbe38aa4736bbc781ffe551"}
Oct 30 11:59:00.833942 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.833561    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:59:00.835325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.835295    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4" event={"ID":"80dc6036-09f8-4b92-a48d-706884607ba8","Type":"ContainerStarted","Data":"4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6"}
Oct 30 11:59:00.835437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.835347    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:59:00.835437 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.835369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4" event={"ID":"80dc6036-09f8-4b92-a48d-706884607ba8","Type":"ContainerStarted","Data":"ae9da01000f27ec19c4dfcbd931fddb021b203fe57da511e39546ede85959561"}
Oct 30 11:59:00.876189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.876138    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk" podStartSLOduration=1.876125534 podStartE2EDuration="1.876125534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:00.874216163 +0000 UTC m=+1744.669100656" watchObservedRunningTime="2026-10-30 11:59:00.876125534 +0000 UTC m=+1744.671010048"
Oct 30 11:59:00.917115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:00.917051    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4" podStartSLOduration=1.91703651 podStartE2EDuration="1.91703651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:00.914715324 +0000 UTC m=+1744.709599824" watchObservedRunningTime="2026-10-30 11:59:00.91703651 +0000 UTC m=+1744.711921004"
Oct 30 11:59:01.866134 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:01.866096    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:02.710502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:02.708470    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:02.724309 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:02.722632    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:59:02.725404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:02.725378    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:59:03.734740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:03.734704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:04.257593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:04.257558    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:04.756873 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:04.756833    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59" podUID="0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" containerName="payload-processing" containerID="cri-o://851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603" gracePeriod=25
Oct 30 11:59:04.790710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:04.790659    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m" podUID="625faff0-053f-455c-8f59-cf8f99b29c53" containerName="payload-pre-processing" containerID="cri-o://b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421" gracePeriod=25
Oct 30 11:59:05.086338 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.086309    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:59:05.112153 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.112121    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:59:05.155323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.155284    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume\") pod \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") "
Oct 30 11:59:05.155552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.155518    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4\" (UniqueName: \"kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4\") pod \"625faff0-053f-455c-8f59-cf8f99b29c53\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") "
Oct 30 11:59:05.155612 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.155586    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume\") pod \"625faff0-053f-455c-8f59-cf8f99b29c53\" (UID: \"625faff0-053f-455c-8f59-cf8f99b29c53\") "
Oct 30 11:59:05.155665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.155647    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb\" (UniqueName: \"kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb\") pod \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\" (UID: \"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096\") "
Oct 30 11:59:05.155748 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.155722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume" pod "0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" (UID: "0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:05.156101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.156078    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:05.157026 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.156310    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume" pod "625faff0-053f-455c-8f59-cf8f99b29c53" (UID: "625faff0-053f-455c-8f59-cf8f99b29c53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:05.158114 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.158086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4" pod "625faff0-053f-455c-8f59-cf8f99b29c53" (UID: "625faff0-053f-455c-8f59-cf8f99b29c53"). InnerVolumeSpecName "kube-api-access-74cd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:05.158710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.158684    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb" pod "0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" (UID: "0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096"). InnerVolumeSpecName "kube-api-access-8flnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:05.257118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.257078    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flnb\" (UniqueName: \"kubernetes.io/projected/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096-kube-api-access-8flnb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:05.257118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.257106    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74cd4\" (UniqueName: \"kubernetes.io/projected/625faff0-053f-455c-8f59-cf8f99b29c53-kube-api-access-74cd4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:05.257118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.257116    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625faff0-053f-455c-8f59-cf8f99b29c53-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:05.738719 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.738687    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:05.794740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.794708    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 11:59:05.795233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.794784    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 11:59:05.878853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.878818    5060 generic.go:356] "Generic (PLEG): container finished" podID="0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" containerID="851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603" exitCode=0
Oct 30 11:59:05.879018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.878965    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"
Oct 30 11:59:05.879608 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.879557    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59" event={"ID":"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096","Type":"ContainerDied","Data":"851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603"}
Oct 30 11:59:05.879745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.879626    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59" event={"ID":"0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096","Type":"ContainerDied","Data":"e61980f98a2121d15ab994968aaa593e20b9413985687ac1a4c8ee4ddb3c7ef6"}
Oct 30 11:59:05.879745 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.879659    5060 scope.go:122] "RemoveContainer" containerID="851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603"
Oct 30 11:59:05.881890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.881371    5060 generic.go:356] "Generic (PLEG): container finished" podID="625faff0-053f-455c-8f59-cf8f99b29c53" containerID="b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421" exitCode=0
Oct 30 11:59:05.881890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.881410    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m" event={"ID":"625faff0-053f-455c-8f59-cf8f99b29c53","Type":"ContainerDied","Data":"b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421"}
Oct 30 11:59:05.881890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.881430    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"
Oct 30 11:59:05.881890 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.881493    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m" event={"ID":"625faff0-053f-455c-8f59-cf8f99b29c53","Type":"ContainerDied","Data":"c1da520b32e4b76790761c6ad5bcaefb525af2a46794ef5790d8452c8bbdb9ba"}
Oct 30 11:59:05.893472 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.893436    5060 scope.go:122] "RemoveContainer" containerID="851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603"
Oct 30 11:59:05.893830 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:59:05.893810    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603\": container with ID starting with 851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603 not found: ID does not exist" containerID="851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603"
Oct 30 11:59:05.893908 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.893839    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603"} err="failed to get container status \"851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603\": rpc error: code = NotFound desc = could not find container \"851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603\": container with ID starting with 851e5036e20ab2bd419568227e9cc8ab8a9e7f74361c8377821e4a870c495603 not found: ID does not exist"
Oct 30 11:59:05.893908 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.893858    5060 scope.go:122] "RemoveContainer" containerID="b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421"
Oct 30 11:59:05.907031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.906994    5060 scope.go:122] "RemoveContainer" containerID="b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421"
Oct 30 11:59:05.907507 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:59:05.907474    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421\": container with ID starting with b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421 not found: ID does not exist" containerID="b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421"
Oct 30 11:59:05.907634 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.907518    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421"} err="failed to get container status \"b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421\": rpc error: code = NotFound desc = could not find container \"b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421\": container with ID starting with b58d212552ba82ffba80847e81805a91df0d0e1e733d247e5544c94ad7028421 not found: ID does not exist"
Oct 30 11:59:05.940895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.940858    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"]
Oct 30 11:59:05.945822 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.945791    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f5339-7df9788c5d-xsh59"]
Oct 30 11:59:05.980565 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.980530    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"]
Oct 30 11:59:05.983583 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:05.983554    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f5339-fd9bb57f-49q2m"]
Oct 30 11:59:06.336790 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:06.336700    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" path="/var/lib/kubelet/pods/0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096/volumes"
Oct 30 11:59:06.337160 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:06.337140    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625faff0-053f-455c-8f59-cf8f99b29c53" path="/var/lib/kubelet/pods/625faff0-053f-455c-8f59-cf8f99b29c53/volumes"
Oct 30 11:59:07.399529 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:07.399485    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:09.176770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:09.176731    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:09.322897 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:09.322859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:09.538416 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:09.538378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:10.246043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:10.246005    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:10.399951 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:10.399917    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:11.119044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:11.119005    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:11.268279 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:11.268235    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:11.493089 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:11.493055    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:11.846124 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:11.846037    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 11:59:11.846294 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:11.846275    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 11:59:12.952697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:12.952621    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:14.724186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:14.724150    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:14.875542 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:14.875506    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:16.813056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:16.813022    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:17.095101 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095029    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"]
Oct 30 11:59:17.095573 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095559    5060 cpu_manager.go:414] "containerMap: removing container" podUID="625faff0-053f-455c-8f59-cf8f99b29c53" containerName="payload-pre-processing"
Oct 30 11:59:17.095635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095574    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625faff0-053f-455c-8f59-cf8f99b29c53" containerName="payload-pre-processing"
Oct 30 11:59:17.095635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095609    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" containerName="payload-processing"
Oct 30 11:59:17.095635 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095613    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" containerName="payload-processing"
Oct 30 11:59:17.095738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095694    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe0f4f8-6b56-46b3-9bfb-dd07b8f50096" containerName="payload-processing"
Oct 30 11:59:17.095738 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.095705    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="625faff0-053f-455c-8f59-cf8f99b29c53" containerName="payload-pre-processing"
Oct 30 11:59:17.099006 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.098989    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.126212 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.126182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.127009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.126263    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwvz6\" (UniqueName: \"kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.127136 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.126913    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.136399 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.136370    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:59:17.136552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.136370    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:59:17.136552 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.136407    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:59:17.137929 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.137901    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:59:17.141430 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.141408    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"]
Oct 30 11:59:17.228142 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.228110    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.228347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.228205    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.228347 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.228231    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwvz6\" (UniqueName: \"kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.228789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.228768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.230683 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.230660    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.254649 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.254611    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwvz6\" (UniqueName: \"kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6\") pod \"maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.413672 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.413603    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:17.720077 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:17.720037    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7affbd62_8d26_40c0_a2a5_4fb8667867ad.slice/crio-a885cc1fe32305d120d761ecb41719101857a60596480c107b66881ca9f6248f WatchSource:0}: Error finding container a885cc1fe32305d120d761ecb41719101857a60596480c107b66881ca9f6248f: Status 404 returned error can't find the container with id a885cc1fe32305d120d761ecb41719101857a60596480c107b66881ca9f6248f
Oct 30 11:59:17.721370 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.721340    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"]
Oct 30 11:59:17.982888 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:17.982838    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6" event={"ID":"7affbd62-8d26-40c0-a2a5-4fb8667867ad","Type":"ContainerStarted","Data":"a885cc1fe32305d120d761ecb41719101857a60596480c107b66881ca9f6248f"}
Oct 30 11:59:18.992547 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:18.992514    5060 generic.go:356] "Generic (PLEG): container finished" podID="7affbd62-8d26-40c0-a2a5-4fb8667867ad" containerID="b50b8f0169df6f61500981df78b619d5ffcce81c3f1e46c92dc31a11e6df9506" exitCode=0
Oct 30 11:59:18.993052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:18.992592    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6" event={"ID":"7affbd62-8d26-40c0-a2a5-4fb8667867ad","Type":"ContainerDied","Data":"b50b8f0169df6f61500981df78b619d5ffcce81c3f1e46c92dc31a11e6df9506"}
Oct 30 11:59:19.557592 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:19.557555    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:19.856890 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:59:19.856797    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:59:20.154864 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.154838    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:20.159310 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.159286    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6\" (UniqueName: \"kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6\") pod \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") "
Oct 30 11:59:20.159436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.159337    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca\") pod \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") "
Oct 30 11:59:20.159526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.159493    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca\") pod \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\" (UID: \"7affbd62-8d26-40c0-a2a5-4fb8667867ad\") "
Oct 30 11:59:20.159931 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.159902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca" pod "7affbd62-8d26-40c0-a2a5-4fb8667867ad" (UID: "7affbd62-8d26-40c0-a2a5-4fb8667867ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:20.160249 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.160231    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:20.161688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.161661    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca" pod "7affbd62-8d26-40c0-a2a5-4fb8667867ad" (UID: "7affbd62-8d26-40c0-a2a5-4fb8667867ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:59:20.161797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.161782    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6" pod "7affbd62-8d26-40c0-a2a5-4fb8667867ad" (UID: "7affbd62-8d26-40c0-a2a5-4fb8667867ad"). InnerVolumeSpecName "kube-api-access-rwvz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:20.260912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.260874    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwvz6\" (UniqueName: \"kubernetes.io/projected/7affbd62-8d26-40c0-a2a5-4fb8667867ad-kube-api-access-rwvz6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:20.260912 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:20.260905    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7affbd62-8d26-40c0-a2a5-4fb8667867ad-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:21.012037 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:21.012000    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"
Oct 30 11:59:21.012220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:21.012001    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6" event={"ID":"7affbd62-8d26-40c0-a2a5-4fb8667867ad","Type":"ContainerDied","Data":"a885cc1fe32305d120d761ecb41719101857a60596480c107b66881ca9f6248f"}
Oct 30 11:59:21.012220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:21.012127    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a885cc1fe32305d120d761ecb41719101857a60596480c107b66881ca9f6248f"
Oct 30 11:59:21.309599 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:21.309494    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:21.456956 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:21.456919    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:23.011080 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:23.011037    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:23.626323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:23.626289    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:23.723827 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:23.723788    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"]
Oct 30 11:59:23.737011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:23.736928    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f5339-nxlfm"]
Oct 30 11:59:23.785614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:23.785570    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:24.325695 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:24.325664    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:24.334997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:24.334969    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513a0739-2c70-4e69-8593-d242f99a77fc" path="/var/lib/kubelet/pods/513a0739-2c70-4e69-8593-d242f99a77fc/volumes"
Oct 30 11:59:25.702254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:25.702215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:27.875197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:27.875162    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:29.656464 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:29.656414    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:30.483534 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:30.483500    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:30.639997 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:30.639960    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:30.913688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:30.910393    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"]
Oct 30 11:59:30.913688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:30.910686    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy" containerID="cri-o://1e4c7babd00909b1d403cab4d34c3a4ee98e3a8350bb80fc881483c9382343a7" gracePeriod=30
Oct 30 11:59:30.921536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:30.916275    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:31.505314 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.505266    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"]
Oct 30 11:59:31.506519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.506489    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7affbd62-8d26-40c0-a2a5-4fb8667867ad" containerName="revoke-keys"
Oct 30 11:59:31.506688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.506674    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7affbd62-8d26-40c0-a2a5-4fb8667867ad" containerName="revoke-keys"
Oct 30 11:59:31.516476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.506936    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7affbd62-8d26-40c0-a2a5-4fb8667867ad" containerName="revoke-keys"
Oct 30 11:59:31.553962 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.553923    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:31.556441 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.556405    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"]
Oct 30 11:59:31.556632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.556545    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.572001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.571974    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:59:31.572001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.571986    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:59:31.606692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.606499    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:59:31.606692 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.606655    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:59:31.665863 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.665828    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48j4\" (UniqueName: \"kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.666053 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.665984    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.666053 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.666012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.767918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.767823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48j4\" (UniqueName: \"kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.768115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.767952    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.768115 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.768014    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.768834 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.768807    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.770809 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.770788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.810568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.810531    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48j4\" (UniqueName: \"kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4\") pod \"maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:31.875797 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:31.875763    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:32.305301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:32.305199    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"]
Oct 30 11:59:33.164487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:33.163737    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf" event={"ID":"67a572a8-3cfa-46dc-af83-e49d6beaaae2","Type":"ContainerStarted","Data":"8314b993fd06ccfc59ae8a6ef226d00193511b021cbc07320aa9451473591d83"}
Oct 30 11:59:33.164487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:33.163829    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf" event={"ID":"67a572a8-3cfa-46dc-af83-e49d6beaaae2","Type":"ContainerStarted","Data":"ad85f00ce5b62937910ddfb3e2f7ed3cea4146024ddde446bab7e5641c046696"}
Oct 30 11:59:33.218568 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:33.218491    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf" podStartSLOduration=2.218443379 podStartE2EDuration="2.218443379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:33.216198999 +0000 UTC m=+1777.011083494" watchObservedRunningTime="2026-10-30 11:59:33.218443379 +0000 UTC m=+1777.013327876"
Oct 30 11:59:33.387754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:33.387664    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:34.174984 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:34.174946    5060 generic.go:356] "Generic (PLEG): container finished" podID="67a572a8-3cfa-46dc-af83-e49d6beaaae2" containerID="8314b993fd06ccfc59ae8a6ef226d00193511b021cbc07320aa9451473591d83" exitCode=0
Oct 30 11:59:34.175186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:34.174998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf" event={"ID":"67a572a8-3cfa-46dc-af83-e49d6beaaae2","Type":"ContainerDied","Data":"8314b993fd06ccfc59ae8a6ef226d00193511b021cbc07320aa9451473591d83"}
Oct 30 11:59:35.215752 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.215715    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:35.395616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.395578    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:35.552423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.552383    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca\") pod \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") "
Oct 30 11:59:35.552680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.552506    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4\" (UniqueName: \"kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4\") pod \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") "
Oct 30 11:59:35.552680 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.552660    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca\") pod \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\" (UID: \"67a572a8-3cfa-46dc-af83-e49d6beaaae2\") "
Oct 30 11:59:35.553001 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.552973    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca" pod "67a572a8-3cfa-46dc-af83-e49d6beaaae2" (UID: "67a572a8-3cfa-46dc-af83-e49d6beaaae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:35.553349 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.553319    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:35.554899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.554859    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4" pod "67a572a8-3cfa-46dc-af83-e49d6beaaae2" (UID: "67a572a8-3cfa-46dc-af83-e49d6beaaae2"). InnerVolumeSpecName "kube-api-access-p48j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:35.555014 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.554909    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca" pod "67a572a8-3cfa-46dc-af83-e49d6beaaae2" (UID: "67a572a8-3cfa-46dc-af83-e49d6beaaae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:59:35.654512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.654476    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a572a8-3cfa-46dc-af83-e49d6beaaae2-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:35.654512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:35.654514    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p48j4\" (UniqueName: \"kubernetes.io/projected/67a572a8-3cfa-46dc-af83-e49d6beaaae2-kube-api-access-p48j4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.203486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.203401    5060 generic.go:356] "Generic (PLEG): container finished" podID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerID="1e4c7babd00909b1d403cab4d34c3a4ee98e3a8350bb80fc881483c9382343a7" exitCode=0
Oct 30 11:59:36.204152 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.203952    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" event={"ID":"ba0f9611-6af9-4819-b351-d8cad17c5866","Type":"ContainerDied","Data":"1e4c7babd00909b1d403cab4d34c3a4ee98e3a8350bb80fc881483c9382343a7"}
Oct 30 11:59:36.204152 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.204035    5060 scope.go:122] "RemoveContainer" containerID="ebd2c0a3260fbb35afa0c7558f42f5485a9d9073f1e564a4ebce5f0cf7fd772e"
Oct 30 11:59:36.208543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.208347    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf" event={"ID":"67a572a8-3cfa-46dc-af83-e49d6beaaae2","Type":"ContainerDied","Data":"ad85f00ce5b62937910ddfb3e2f7ed3cea4146024ddde446bab7e5641c046696"}
Oct 30 11:59:36.208543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.208398    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad85f00ce5b62937910ddfb3e2f7ed3cea4146024ddde446bab7e5641c046696"
Oct 30 11:59:36.208543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.208481    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"
Oct 30 11:59:36.262003 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.261973    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:59:36.364950 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.364919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.364970    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.364994    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365069    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365158 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365142    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365182    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365208    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.365505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365280    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token\") pod \"ba0f9611-6af9-4819-b351-d8cad17c5866\" (UID: \"ba0f9611-6af9-4819-b351-d8cad17c5866\") "
Oct 30 11:59:36.366221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.365946    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:36.366879 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.366858    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:36.367000 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.366932    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:36.367116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.367072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:36.367116 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.367098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:36.368810 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.368780    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:36.369052 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.369029    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:36.369564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.369426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:59:36.370893 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.370764    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf" pod "ba0f9611-6af9-4819-b351-d8cad17c5866" (UID: "ba0f9611-6af9-4819-b351-d8cad17c5866"). InnerVolumeSpecName "kube-api-access-mhlkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:36.466600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466566    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.466600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466594    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.466600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466602    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.466600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466610    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.466600 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466618    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.467056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466627    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.467056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466638    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlkf\" (UniqueName: \"kubernetes.io/projected/ba0f9611-6af9-4819-b351-d8cad17c5866-kube-api-access-mhlkf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.467056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466647    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9611-6af9-4819-b351-d8cad17c5866-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.467056 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.466657    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9611-6af9-4819-b351-d8cad17c5866-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:36.606269 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.606154    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"]
Oct 30 11:59:36.606540 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.606432    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" containerID="cri-o://d9c9e521dde33be9355f6b718005b26d67b439a75e6734d391ab008d603866c0" gracePeriod=30
Oct 30 11:59:36.620508 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.620436    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"]
Oct 30 11:59:36.620758 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:36.620711    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" containerID="cri-o://63bfa5af53e8d15d473ab190e63bb2a86897040b6770cac65aa01efebe805f5b" gracePeriod=30
Oct 30 11:59:37.089824 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.088726    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:37.172379 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.172350    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"]
Oct 30 11:59:37.172960 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.172946    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy"
Oct 30 11:59:37.173011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.172962    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy"
Oct 30 11:59:37.173011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.172985    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy"
Oct 30 11:59:37.173011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.172990    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy"
Oct 30 11:59:37.173011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.173002    5060 cpu_manager.go:414] "containerMap: removing container" podUID="67a572a8-3cfa-46dc-af83-e49d6beaaae2" containerName="revoke-keys"
Oct 30 11:59:37.173011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.173008    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a572a8-3cfa-46dc-af83-e49d6beaaae2" containerName="revoke-keys"
Oct 30 11:59:37.173172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.173093    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy"
Oct 30 11:59:37.173172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.173102    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a572a8-3cfa-46dc-af83-e49d6beaaae2" containerName="revoke-keys"
Oct 30 11:59:37.173172 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.173114    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" containerName="istio-proxy"
Oct 30 11:59:37.176375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.176356    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.214351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.208643    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"]
Oct 30 11:59:37.224972 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.224932    5060 generic.go:356] "Generic (PLEG): container finished" podID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerID="5aef57c1ff5eac80ca5883b31cfeca72c410655664574ae1f7a656d5186cd277" exitCode=0
Oct 30 11:59:37.225254 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.225224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" event={"ID":"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5","Type":"ContainerDied","Data":"5aef57c1ff5eac80ca5883b31cfeca72c410655664574ae1f7a656d5186cd277"}
Oct 30 11:59:37.225746 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.225726    5060 scope.go:122] "RemoveContainer" containerID="5aef57c1ff5eac80ca5883b31cfeca72c410655664574ae1f7a656d5186cd277"
Oct 30 11:59:37.228152 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.228114    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:37.228301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.228216    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:37.228301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.228284    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:37.228440 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.228410    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p" event={"ID":"ba0f9611-6af9-4819-b351-d8cad17c5866","Type":"ContainerDied","Data":"b1208add0412a0ee4928cf402b7c3aeec475835bf34773f3ae71d5cb76ee805d"}
Oct 30 11:59:37.228543 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.228501    5060 scope.go:122] "RemoveContainer" containerID="1e4c7babd00909b1d403cab4d34c3a4ee98e3a8350bb80fc881483c9382343a7"
Oct 30 11:59:37.229189 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.228836    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"
Oct 30 11:59:37.277257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64f9q\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277350    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277413    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277528    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277619    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277725    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277754    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.277860 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.277828    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.319164 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.319124    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"]
Oct 30 11:59:37.325247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.325214    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f5339-openshift-default-976c686df-gw75p"]
Oct 30 11:59:37.379220 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379185    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379277    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379342    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379431 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379371    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379496    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64f9q\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379622 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379583    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379633    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379789 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379661    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.379918 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.379890    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.380260 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.380238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.380432 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.380410    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.380548 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.380525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.380806 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.380785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.382750 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.382730    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.385631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.385578    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.405829 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.405794    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64f9q\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.406892 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.406780    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.542228 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.542187    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:37.881676 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:37.881646    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb048634_eb56_4b15_9cd0_cc65cecf3a62.slice/crio-d8cb7e7b467b958289c2e090ae49affdefdf9e678d1c8c83d2b8007b3b4eaa45 WatchSource:0}: Error finding container d8cb7e7b467b958289c2e090ae49affdefdf9e678d1c8c83d2b8007b3b4eaa45: Status 404 returned error can't find the container with id d8cb7e7b467b958289c2e090ae49affdefdf9e678d1c8c83d2b8007b3b4eaa45
Oct 30 11:59:37.883691 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.883669    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"]
Oct 30 11:59:37.884796 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.884754    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:37.884915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.884841    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:37.884915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:37.884882    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:38.238287 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.238251    5060 generic.go:356] "Generic (PLEG): container finished" podID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerID="e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237" exitCode=0
Oct 30 11:59:38.238531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.238327    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" event={"ID":"d6b9dfb1-f962-4e38-8c48-989fb1478455","Type":"ContainerDied","Data":"e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"}
Oct 30 11:59:38.238853 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.238834    5060 scope.go:122] "RemoveContainer" containerID="e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"
Oct 30 11:59:38.241156 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.241122    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:38.241278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.241215    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:38.241278 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.241262    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:38.242331 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.242310    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" event={"ID":"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5","Type":"ContainerStarted","Data":"3cadf045920aac80e7f67d4b67d82219336dd5bffcc38d05d52e4c82183653bf"}
Oct 30 11:59:38.244439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.244374    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" event={"ID":"eb048634-eb56-4b15-9cd0-cc65cecf3a62","Type":"ContainerStarted","Data":"1733dc5dd1f8afca9f911cb5ce0ea71662e32f860081365df3009bd4289e9788"}
Oct 30 11:59:38.244439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.244414    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" event={"ID":"eb048634-eb56-4b15-9cd0-cc65cecf3a62","Type":"ContainerStarted","Data":"d8cb7e7b467b958289c2e090ae49affdefdf9e678d1c8c83d2b8007b3b4eaa45"}
Oct 30 11:59:38.251841 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.251821    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:59:38.252979 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.252956    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 11:59:38.253081 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.253003    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 11:59:38.349740 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.349710    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0f9611-6af9-4819-b351-d8cad17c5866" path="/var/lib/kubelet/pods/ba0f9611-6af9-4819-b351-d8cad17c5866/volumes"
Oct 30 11:59:38.354646 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.354598    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podStartSLOduration=1.354584076 podStartE2EDuration="1.354584076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:38.338218971 +0000 UTC m=+1782.133103467" watchObservedRunningTime="2026-10-30 11:59:38.354584076 +0000 UTC m=+1782.149468569"
Oct 30 11:59:38.544588 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.543108    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:38.545351 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.545315    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:38.545616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.545588    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:38.993478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:38.990916    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:39.265415 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.265321    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 11:59:39.265742 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.265709    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 11:59:39.275694 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.274657    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" event={"ID":"d6b9dfb1-f962-4e38-8c48-989fb1478455","Type":"ContainerStarted","Data":"a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4"}
Oct 30 11:59:39.298476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.288200    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"]
Oct 30 11:59:39.341611 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.337217    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"]
Oct 30 11:59:39.665536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.650062    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:59:39.719478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.692672    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:39.719478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.692770    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:39.864231 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.864164    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:59:39.936952 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.936860    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 11:59:39.937247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.937210    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 11:59:39.949214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:39.949175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:40.162477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.160069    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:40.251895 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.251862    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 11:59:40.252208 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.252170    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 11:59:40.554999 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.554901    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:40.555282 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.555252    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:40.824656 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.824556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:40.881576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.863606    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 11:59:40.881576 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:40.863673    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 11:59:41.024841 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.024810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:41.193192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.193038    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:41.259482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.251864    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 11:59:41.259482 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.251924    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 11:59:41.418777 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.418738    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:59:41.419352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.419078    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:59:41.561494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.551828    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:41.561494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.551891    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:41.825059 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.822231    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:41.871614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.869627    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 11:59:41.871614 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.869729    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 11:59:41.982657 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.982219    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"]
Oct 30 11:59:41.983511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:41.983330    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:42.043480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.034613    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:42.044216 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.044185    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5da940-kjtf6"]
Oct 30 11:59:42.070090 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.070048    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 11:59:42.072328 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.072293    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:42.188502 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.173543    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:59:42.238512 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.236126    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:42.254398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.254362    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 11:59:42.254707 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.254674    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 11:59:42.372531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.352651    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:42.486481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.484849    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7affbd62-8d26-40c0-a2a5-4fb8667867ad" path="/var/lib/kubelet/pods/7affbd62-8d26-40c0-a2a5-4fb8667867ad/volumes"
Oct 30 11:59:42.538701 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.536998    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:42.577621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.577578    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerID="63bfa5af53e8d15d473ab190e63bb2a86897040b6770cac65aa01efebe805f5b" exitCode=0
Oct 30 11:59:42.578018 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.577994    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" event={"ID":"0ed8feed-bf08-429d-9a7c-bd8cadbee54e","Type":"ContainerDied","Data":"63bfa5af53e8d15d473ab190e63bb2a86897040b6770cac65aa01efebe805f5b"}
Oct 30 11:59:42.631902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.631805    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:42.631902 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.631862    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:42.646812 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.646767    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:42.693494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.676199    5060 generic.go:356] "Generic (PLEG): container finished" podID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerID="d9c9e521dde33be9355f6b718005b26d67b439a75e6734d391ab008d603866c0" exitCode=0
Oct 30 11:59:42.693494 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.676437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" event={"ID":"a9a37abf-a6cc-4e03-8490-e1c9da962c87","Type":"ContainerDied","Data":"d9c9e521dde33be9355f6b718005b26d67b439a75e6734d391ab008d603866c0"}
Oct 30 11:59:42.703516 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.700088    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:42.760795 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.760765    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:59:42.847490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.841920    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:42.864911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.864093    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 11:59:42.864911 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.864151    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 11:59:42.917535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.900958    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.948590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.948541    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.973477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.948849    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.973837 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.973809    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.974002 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.973987    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.974210 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.974192    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.974341 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.974327    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.974546 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.974529    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.974685 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.974671    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert\") pod \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\" (UID: \"0ed8feed-bf08-429d-9a7c-bd8cadbee54e\") "
Oct 30 11:59:42.992637 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.921553    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:42.992886 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.992579    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:42.993236 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.993218    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:42.993603 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:42.993586    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:43.025486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.023015    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:43.050095 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.050056    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:43.050230 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.050164    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.050590 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.050569    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.126576    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.126973    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.127623    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.127708    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.130131    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.130157    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.130172    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.130186    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.133499 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.130199    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.182489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.176825    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.198487 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.197726    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:59:43.286505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.278632    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:43.286505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.278763    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:43.286505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.281605    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.286505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.281634    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.286505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.281688    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.355479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.350440    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 11:59:43.378478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.355778    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.378478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.366616    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "kube-api-access-kc998". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:43.475572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.396870    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo" pod "0ed8feed-bf08-429d-9a7c-bd8cadbee54e" (UID: "0ed8feed-bf08-429d-9a7c-bd8cadbee54e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:59:43.475572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.396980    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:59:43.475572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.398439    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc998\" (UniqueName: \"kubernetes.io/projected/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-kube-api-access-kc998\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.475572 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.421850    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed8feed-bf08-429d-9a7c-bd8cadbee54e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:43.508478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.507984    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 11:59:43.508478 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.508038    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 11:59:43.541654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.541604    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:59:43.546011 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.545982    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:43.551605 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.551552    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:43.632480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.632327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:43.651569 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.649578    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.685477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.651884    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:59:43.686305 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.685804    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.867536 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.866029    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:43.913104 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.913029    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 11:59:43.913945 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.913874    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:59:43.966551 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.966507    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.083006 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.082949    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.083245 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.083229    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.083385 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.083371    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.083597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.083581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.084564 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.084536    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.084771 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.084756    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.093703 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.093666    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.219505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.205174    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.249562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.234010    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 11:59:44.249891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.141228    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:44.289013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.288967    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6" podUID="a758c044-bc32-4683-8da5-b5351a394fe1" containerName="payload-processing" containerID="cri-o://e0403a324eb6b645564d91dad85094660e268e9aa9e8facc60d22ee24afd3da1" gracePeriod=25
Oct 30 11:59:44.359555 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:43.999212    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:44.359785 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.001985    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 11:59:44.359941 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.359914    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 11:59:44.360043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.061942    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:44.360433 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.302184    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 11:59:44.373879 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.373428    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "kube-api-access-np72w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.349872    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.094237    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.117393    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.140406    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.188980    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.191780    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.391162    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.354871    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.391348    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"
Oct 30 11:59:44.407843 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.401155    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:59:44.568170 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.562664    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:44.615526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.594380    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 11:59:44.615526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.594474    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 11:59:44.615526 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.596199    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb" podUID="06eb3845-8e72-4f96-b978-8e0280920fdd" containerName="payload-pre-processing" containerID="cri-o://6c5bff1ab1f71d2133c28a634380333722bfa271d2dbb7f5024d3068ec4fe93a" gracePeriod=25
Oct 30 11:59:44.631340 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.616168    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.676643    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\") pod \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\" (UID: \"a9a37abf-a6cc-4e03-8490-e1c9da962c87\") "
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677304    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677336    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np72w\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-kube-api-access-np72w\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677354    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677372    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677387    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677407    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677420    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677434    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:44.677523    5060 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a9a37abf-a6cc-4e03-8490-e1c9da962c87/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 30 11:59:44.715500 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.677544    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy" pod "a9a37abf-a6cc-4e03-8490-e1c9da962c87" (UID: "a9a37abf-a6cc-4e03-8490-e1c9da962c87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:44.772257 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.772217    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt" event={"ID":"a9a37abf-a6cc-4e03-8490-e1c9da962c87","Type":"ContainerDied","Data":"8c14318ba753c7f274be6e567ff7991e6eb98b9ac4f4b4a4d6ad7748cc23bab5"}
Oct 30 11:59:44.772639 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.772617    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 11:59:44.772845 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.772832    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 11:59:44.773027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.773014    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 11:59:44.775308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.773268    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk" event={"ID":"0ed8feed-bf08-429d-9a7c-bd8cadbee54e","Type":"ContainerDied","Data":"2e54ef011d79665dc9847b1a8bb0de06e1108eca7f8b7b85819f85a09b976328"}
Oct 30 11:59:44.775308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.773334    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"]
Oct 30 11:59:44.803032 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.802947    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a37abf-a6cc-4e03-8490-e1c9da962c87-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:44.805607 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.805578    5060 scope.go:122] "RemoveContainer" containerID="d9c9e521dde33be9355f6b718005b26d67b439a75e6734d391ab008d603866c0"
Oct 30 11:59:44.815900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.807745    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy"
Oct 30 11:59:44.816099 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.816083    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy"
Oct 30 11:59:44.816229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.816218    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy"
Oct 30 11:59:44.816299 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.816289    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy"
Oct 30 11:59:44.825905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.819143    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" containerName="istio-proxy"
Oct 30 11:59:44.825905 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.819174    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" containerName="istio-proxy"
Oct 30 11:59:44.864154 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.864122    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 11:59:44.864410 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.864388    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 11:59:44.895490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.877665    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"]
Oct 30 11:59:44.895490 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.877801    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:44.917654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.896616    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-eb5531-openshift-default-dockercfg-fn4kw\""
Oct 30 11:59:44.972658 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:44.970491    5060 scope.go:122] "RemoveContainer" containerID="63bfa5af53e8d15d473ab190e63bb2a86897040b6770cac65aa01efebe805f5b"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016480    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016639    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016698    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016730    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016878    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.016989    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtldg\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.019479 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.017034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.098631 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.098545    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"]
Oct 30 11:59:45.121880 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.121846    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rjwvt"]
Oct 30 11:59:45.123027 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.122985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.123243 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.123224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.123515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.123494    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.123697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.123677    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.123830 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.123815    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.124021 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.123988    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.124145 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.124131    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.124272 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.124258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtldg\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.124404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.124391    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.126343 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.126288    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.126919 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.126893    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.127439 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.127413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.129301 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.129280    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.129697 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.129680    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.141323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.141249    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.146486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.143042    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.189044 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.188891    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.238308 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.222216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtldg\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg\") pod \"e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") " pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.256192 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.249403    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:45.294155 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.288854    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"]
Oct 30 11:59:45.311480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.294362    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-rcqpk"]
Oct 30 11:59:45.322613 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.322569    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:59:45.390404 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.390312    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 11:59:45.417225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.417013    5060 generic.go:356] "Generic (PLEG): container finished" podID="a758c044-bc32-4683-8da5-b5351a394fe1" containerID="e0403a324eb6b645564d91dad85094660e268e9aa9e8facc60d22ee24afd3da1" exitCode=0
Oct 30 11:59:45.417225 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.417160    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6" event={"ID":"a758c044-bc32-4683-8da5-b5351a394fe1","Type":"ContainerDied","Data":"e0403a324eb6b645564d91dad85094660e268e9aa9e8facc60d22ee24afd3da1"}
Oct 30 11:59:45.421706 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.421672    5060 generic.go:356] "Generic (PLEG): container finished" podID="06eb3845-8e72-4f96-b978-8e0280920fdd" containerID="6c5bff1ab1f71d2133c28a634380333722bfa271d2dbb7f5024d3068ec4fe93a" exitCode=0
Oct 30 11:59:45.422616 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.422583    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb" event={"ID":"06eb3845-8e72-4f96-b978-8e0280920fdd","Type":"ContainerDied","Data":"6c5bff1ab1f71d2133c28a634380333722bfa271d2dbb7f5024d3068ec4fe93a"}
Oct 30 11:59:45.446221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.446195    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:59:45.557364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.557337    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:59:45.582197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.580551    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume\") pod \"06eb3845-8e72-4f96-b978-8e0280920fdd\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") "
Oct 30 11:59:45.582197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.580755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t\" (UniqueName: \"kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t\") pod \"06eb3845-8e72-4f96-b978-8e0280920fdd\" (UID: \"06eb3845-8e72-4f96-b978-8e0280920fdd\") "
Oct 30 11:59:45.582197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.580945    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume" pod "06eb3845-8e72-4f96-b978-8e0280920fdd" (UID: "06eb3845-8e72-4f96-b978-8e0280920fdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:45.582197 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.581352    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06eb3845-8e72-4f96-b978-8e0280920fdd-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:45.584232 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.584209    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t" pod "06eb3845-8e72-4f96-b978-8e0280920fdd" (UID: "06eb3845-8e72-4f96-b978-8e0280920fdd"). InnerVolumeSpecName "kube-api-access-g6h7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:45.687913 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.687877    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume\") pod \"a758c044-bc32-4683-8da5-b5351a394fe1\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") "
Oct 30 11:59:45.699322 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.699246    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh\" (UniqueName: \"kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh\") pod \"a758c044-bc32-4683-8da5-b5351a394fe1\" (UID: \"a758c044-bc32-4683-8da5-b5351a394fe1\") "
Oct 30 11:59:45.700480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.700434    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h7t\" (UniqueName: \"kubernetes.io/projected/06eb3845-8e72-4f96-b978-8e0280920fdd-kube-api-access-g6h7t\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:45.700665 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.699104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume" pod "a758c044-bc32-4683-8da5-b5351a394fe1" (UID: "a758c044-bc32-4683-8da5-b5351a394fe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:45.726527 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.715811    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh" pod "a758c044-bc32-4683-8da5-b5351a394fe1" (UID: "a758c044-bc32-4683-8da5-b5351a394fe1"). InnerVolumeSpecName "kube-api-access-tb7wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:45.781511 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.777727    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"]
Oct 30 11:59:45.800495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.798858    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:45.800495 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.800218    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:45.804325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.803101    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c044-bc32-4683-8da5-b5351a394fe1-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:45.804325 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.803126    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7wh\" (UniqueName: \"kubernetes.io/projected/a758c044-bc32-4683-8da5-b5351a394fe1-kube-api-access-tb7wh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:45.807621 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.807598    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 11:59:45.809889 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:45.809839    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dda6dab_3eed_443f_9a4a_b5d7e3fd3de0.slice/crio-b83a7d8c728b89c4b2961b0c94eff3146ac0c4e6575ea93923507c21081bed89 WatchSource:0}: Error finding container b83a7d8c728b89c4b2961b0c94eff3146ac0c4e6575ea93923507c21081bed89: Status 404 returned error can't find the container with id b83a7d8c728b89c4b2961b0c94eff3146ac0c4e6575ea93923507c21081bed89
Oct 30 11:59:45.815515 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.815379    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:45.827525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.815739    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:45.827525 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.815803    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:45.884289 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.884256    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:45.971352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.969903    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:59:45.971352 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.971292    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:59:45.974904 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:45.974846    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 11:59:46.368445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.365481    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed8feed-bf08-429d-9a7c-bd8cadbee54e" path="/var/lib/kubelet/pods/0ed8feed-bf08-429d-9a7c-bd8cadbee54e/volumes"
Oct 30 11:59:46.368445 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.366246    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a37abf-a6cc-4e03-8490-e1c9da962c87" path="/var/lib/kubelet/pods/a9a37abf-a6cc-4e03-8490-e1c9da962c87/volumes"
Oct 30 11:59:46.431057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.431029    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"
Oct 30 11:59:46.431057 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.431046    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6" event={"ID":"a758c044-bc32-4683-8da5-b5351a394fe1","Type":"ContainerDied","Data":"2f8e21974ef6b5edf5042783781361e2dc7c261147154b3714f3973af10cadd0"}
Oct 30 11:59:46.431307 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.431127    5060 scope.go:122] "RemoveContainer" containerID="e0403a324eb6b645564d91dad85094660e268e9aa9e8facc60d22ee24afd3da1"
Oct 30 11:59:46.433434 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.433406    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" event={"ID":"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0","Type":"ContainerStarted","Data":"dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"}
Oct 30 11:59:46.433770 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.433751    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" event={"ID":"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0","Type":"ContainerStarted","Data":"b83a7d8c728b89c4b2961b0c94eff3146ac0c4e6575ea93923507c21081bed89"}
Oct 30 11:59:46.435418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.435362    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"
Oct 30 11:59:46.435418 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.435368    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb" event={"ID":"06eb3845-8e72-4f96-b978-8e0280920fdd","Type":"ContainerDied","Data":"3b08d2e99598138a1531cd2a5494ec3d96db10356715e609171c367ce4de9a42"}
Oct 30 11:59:46.443443 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.443408    5060 scope.go:122] "RemoveContainer" containerID="6c5bff1ab1f71d2133c28a634380333722bfa271d2dbb7f5024d3068ec4fe93a"
Oct 30 11:59:46.497412 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.497353    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podStartSLOduration=3.4973341270000002 podStartE2EDuration="3.497334127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:46.493595303 +0000 UTC m=+1790.288479797" watchObservedRunningTime="2026-10-30 11:59:46.497334127 +0000 UTC m=+1790.292218622"
Oct 30 11:59:46.532384 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.532357    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"]
Oct 30 11:59:46.536840 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.536815    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-747bcf8766-hv7r6"]
Oct 30 11:59:46.593357 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.593322    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"]
Oct 30 11:59:46.596470 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:46.596428    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-56fdd77d6c-dfpbb"]
Oct 30 11:59:47.250055 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:47.250021    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:47.255409 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:47.255381    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:47.447215 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:47.447171    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:47.448131 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:47.448108    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 11:59:47.589731 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:47.589633    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:48.262270 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.262235    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:48.336625 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.336591    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06eb3845-8e72-4f96-b978-8e0280920fdd" path="/var/lib/kubelet/pods/06eb3845-8e72-4f96-b978-8e0280920fdd/volumes"
Oct 30 11:59:48.337152 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.337135    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a758c044-bc32-4683-8da5-b5351a394fe1" path="/var/lib/kubelet/pods/a758c044-bc32-4683-8da5-b5351a394fe1/volumes"
Oct 30 11:59:48.395968 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.395935    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"]
Oct 30 11:59:48.409436 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.409405    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e9b2dd-cqnjf"]
Oct 30 11:59:48.415206 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.415171    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:48.986364 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:48.986329    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:49.428970 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:49.428933    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:49.569317 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:49.569279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:49.868187 ip-10-0-142-105 kubenswrapper[5060]: E1030 11:59:49.868141    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 11:59:49.893049 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:49.893013    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:50.253083 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.253044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:50.337903 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.337872    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a572a8-3cfa-46dc-af83-e49d6beaaae2" path="/var/lib/kubelet/pods/67a572a8-3cfa-46dc-af83-e49d6beaaae2/volumes"
Oct 30 11:59:50.338186 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338172    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"]
Oct 30 11:59:50.338661 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338649    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a758c044-bc32-4683-8da5-b5351a394fe1" containerName="payload-processing"
Oct 30 11:59:50.338704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338662    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a758c044-bc32-4683-8da5-b5351a394fe1" containerName="payload-processing"
Oct 30 11:59:50.338704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338680    5060 cpu_manager.go:414] "containerMap: removing container" podUID="06eb3845-8e72-4f96-b978-8e0280920fdd" containerName="payload-pre-processing"
Oct 30 11:59:50.338704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338686    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06eb3845-8e72-4f96-b978-8e0280920fdd" containerName="payload-pre-processing"
Oct 30 11:59:50.338808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338765    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a758c044-bc32-4683-8da5-b5351a394fe1" containerName="payload-processing"
Oct 30 11:59:50.338808 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.338776    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="06eb3845-8e72-4f96-b978-8e0280920fdd" containerName="payload-pre-processing"
Oct 30 11:59:50.343315 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.343295    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.356293 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.356266    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"]
Oct 30 11:59:50.370963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.369687    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 11:59:50.370963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.369693    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:59:50.370963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.369842    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:59:50.370963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.370142    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.370963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.370232    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nph\" (UniqueName: \"kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.370963 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.370487    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.372597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.372570    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 11:59:50.471205 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.471176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.471739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.471247    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nph\" (UniqueName: \"kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.471739 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.471347    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.471995 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.471973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.473754 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.473738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.495233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.495207    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nph\" (UniqueName: \"kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph\") pod \"maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.555118 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.555013    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:50.659233 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.659198    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:50.969916 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:50.969888    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01b5e2c_97c3_495b_b02f_5fc051b28aaa.slice/crio-cad0496e1c0253e542628047330c13232bbc31b6f2744b2e379bb3f25a2be6b7 WatchSource:0}: Error finding container cad0496e1c0253e542628047330c13232bbc31b6f2744b2e379bb3f25a2be6b7: Status 404 returned error can't find the container with id cad0496e1c0253e542628047330c13232bbc31b6f2744b2e379bb3f25a2be6b7
Oct 30 11:59:50.971249 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:50.971221    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"]
Oct 30 11:59:51.482446 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:51.482406    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5" event={"ID":"d01b5e2c-97c3-495b-b02f-5fc051b28aaa","Type":"ContainerStarted","Data":"edb2364ded11cde72c7274ab93cb50b488e48ac7190401e56fa01a9a527cb493"}
Oct 30 11:59:51.482857 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:51.482501    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5" event={"ID":"d01b5e2c-97c3-495b-b02f-5fc051b28aaa","Type":"ContainerStarted","Data":"cad0496e1c0253e542628047330c13232bbc31b6f2744b2e379bb3f25a2be6b7"}
Oct 30 11:59:51.522928 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:51.522875    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5" podStartSLOduration=1.522856762 podStartE2EDuration="1.522856762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:51.521551692 +0000 UTC m=+1795.316436185" watchObservedRunningTime="2026-10-30 11:59:51.522856762 +0000 UTC m=+1795.317741255"
Oct 30 11:59:51.733164 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:51.733127    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:52.175060 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.174976    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"]
Oct 30 11:59:52.179229 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.179203    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.197899 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.197864    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4093c5-openshift-default-dockercfg-lmvg2\""
Oct 30 11:59:52.237491 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.237442    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"]
Oct 30 11:59:52.307481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.307125    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.307481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.307203    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.307481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.307247    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.307481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.307307    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.307481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.307378    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.307481 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.307421    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.319480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.315500    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.319480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.315647    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2wr\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.319480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.315703    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.359173 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.359139    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:52.419246 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419202    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419306    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419423 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419517    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2wr\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419593 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419564    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419597    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419660    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419702 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419686    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.419859 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.419700    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.420139 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.420110    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.420598 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.420574    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.420741 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.420718    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.421137 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.421115    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.422597 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.422558    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.423247 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.423225    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.444188 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.444114    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2wr\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.452767 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.444580    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token\") pod \"e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") " pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.493688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.493625    5060 generic.go:356] "Generic (PLEG): container finished" podID="d01b5e2c-97c3-495b-b02f-5fc051b28aaa" containerID="edb2364ded11cde72c7274ab93cb50b488e48ac7190401e56fa01a9a527cb493" exitCode=0
Oct 30 11:59:52.494095 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.493825    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5" event={"ID":"d01b5e2c-97c3-495b-b02f-5fc051b28aaa","Type":"ContainerDied","Data":"edb2364ded11cde72c7274ab93cb50b488e48ac7190401e56fa01a9a527cb493"}
Oct 30 11:59:52.497664 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.497631    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:52.513013 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.512979    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:52.661444 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.661410    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:52.827628 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:52.827597    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1a886e_4f3a_4d6b_a51a_5238a17bb9ea.slice/crio-ebd116c488a1f51d3869c77a2f502e0921e271ff96e8285fbefb3cb5b01eb430 WatchSource:0}: Error finding container ebd116c488a1f51d3869c77a2f502e0921e271ff96e8285fbefb3cb5b01eb430: Status 404 returned error can't find the container with id ebd116c488a1f51d3869c77a2f502e0921e271ff96e8285fbefb3cb5b01eb430
Oct 30 11:59:52.828425 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.828396    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"]
Oct 30 11:59:52.830772 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.830734    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:52.830861 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.830832    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:52.830898 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:52.830877    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 11:59:53.505427 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.505388    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" event={"ID":"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea","Type":"ContainerStarted","Data":"beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"}
Oct 30 11:59:53.505983 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.505469    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" event={"ID":"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea","Type":"ContainerStarted","Data":"ebd116c488a1f51d3869c77a2f502e0921e271ff96e8285fbefb3cb5b01eb430"}
Oct 30 11:59:53.568414 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.568350    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podStartSLOduration=1.56832846 podStartE2EDuration="1.56832846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:53.565882844 +0000 UTC m=+1797.360767376" watchObservedRunningTime="2026-10-30 11:59:53.56832846 +0000 UTC m=+1797.363212955"
Oct 30 11:59:53.666126 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.666098    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:53.736818 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.736783    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca\") pod \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") "
Oct 30 11:59:53.737016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.736908    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph\" (UniqueName: \"kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph\") pod \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") "
Oct 30 11:59:53.737016 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.736948    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca\") pod \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\" (UID: \"d01b5e2c-97c3-495b-b02f-5fc051b28aaa\") "
Oct 30 11:59:53.737447 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.737364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca" pod "d01b5e2c-97c3-495b-b02f-5fc051b28aaa" (UID: "d01b5e2c-97c3-495b-b02f-5fc051b28aaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:53.739927 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.739900    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca" pod "d01b5e2c-97c3-495b-b02f-5fc051b28aaa" (UID: "d01b5e2c-97c3-495b-b02f-5fc051b28aaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:59:53.740031 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.739934    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph" pod "d01b5e2c-97c3-495b-b02f-5fc051b28aaa" (UID: "d01b5e2c-97c3-495b-b02f-5fc051b28aaa"). InnerVolumeSpecName "kube-api-access-c6nph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:53.838535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.838431    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:53.838535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.838489    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nph\" (UniqueName: \"kubernetes.io/projected/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-kube-api-access-c6nph\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:53.838535 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:53.838505    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01b5e2c-97c3-495b-b02f-5fc051b28aaa-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 11:59:54.421363 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.421326    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:54.498043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.498010    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:54.504354 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.504326    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:54.516632 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.516599    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"
Oct 30 11:59:54.517144 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.516650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5" event={"ID":"d01b5e2c-97c3-495b-b02f-5fc051b28aaa","Type":"ContainerDied","Data":"cad0496e1c0253e542628047330c13232bbc31b6f2744b2e379bb3f25a2be6b7"}
Oct 30 11:59:54.517144 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.516705    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad0496e1c0253e542628047330c13232bbc31b6f2744b2e379bb3f25a2be6b7"
Oct 30 11:59:54.517144 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.517047    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:54.518285 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:54.518266    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 11:59:55.693900 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:55.693855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:55.842214 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:55.842180    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:56.000949 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.000914    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:56.164693 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.164650    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:56.343813 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.339583    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:56.452943 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.452909    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"]
Oct 30 11:59:56.453647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.453629    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d01b5e2c-97c3-495b-b02f-5fc051b28aaa" containerName="revoke-keys"
Oct 30 11:59:56.453647 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.453647    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01b5e2c-97c3-495b-b02f-5fc051b28aaa" containerName="revoke-keys"
Oct 30 11:59:56.453778 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.453728    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01b5e2c-97c3-495b-b02f-5fc051b28aaa" containerName="revoke-keys"
Oct 30 11:59:56.457401 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.457376    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.490779 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.490747    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"]
Oct 30 11:59:56.529043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.525421    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-48e1f1-dockercfg-wskhz\""
Oct 30 11:59:56.529043 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.525639    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-48e1f1\""
Oct 30 11:59:56.574443 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.574412    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmkt\" (UniqueName: \"kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt\") pod \"payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.574710 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.574690    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.675714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.675622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.675714 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.675697    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmkt\" (UniqueName: \"kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt\") pod \"payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.680475 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.676559    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.711323 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.711285    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmkt\" (UniqueName: \"kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt\") pod \"payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:56.718316 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.718291    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:59:56.738539 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.738517    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 11:59:56.775398 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:56.775362    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:57.115936 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.115611    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"]
Oct 30 11:59:57.128116 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:57.128075    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6500d9_1e72_4ea3_a3fb_b51aed8c8123.slice/crio-3f050e05af498eb8ccc20e8aab53806bd6388152ad0594018653a01a6f7d479a WatchSource:0}: Error finding container 3f050e05af498eb8ccc20e8aab53806bd6388152ad0594018653a01a6f7d479a: Status 404 returned error can't find the container with id 3f050e05af498eb8ccc20e8aab53806bd6388152ad0594018653a01a6f7d479a
Oct 30 11:59:57.284514 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.277026    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:57.459986 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.459955    5060 scope.go:122] "RemoveContainer" containerID="b9c3a4d73e8fe8f2f194a6fa8e6dee722155ad8a39a5402b9b04c359f4229679"
Oct 30 11:59:57.471802 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.471782    5060 scope.go:122] "RemoveContainer" containerID="cf3af293458df1dcd08af229a2883f6d448764bf37c1d32ffb467a73496417de"
Oct 30 11:59:57.508681 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.508524    5060 scope.go:122] "RemoveContainer" containerID="4aac1330c8d331490f0a7fc92cb38e3d80213668a44ae1a133b2556f1f126598"
Oct 30 11:59:57.559708 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.559675    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz" event={"ID":"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123","Type":"ContainerStarted","Data":"a189a5cbeaaa53353ef5864deb2434fb7426c45457b70376e69da352edfa0d3f"}
Oct 30 11:59:57.559842 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.559742    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz" event={"ID":"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123","Type":"ContainerStarted","Data":"3f050e05af498eb8ccc20e8aab53806bd6388152ad0594018653a01a6f7d479a"}
Oct 30 11:59:57.559965 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.559946    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 11:59:57.918832 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:57.918710    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:58.048288 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.048227    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz" podStartSLOduration=2.048207386 podStartE2EDuration="2.048207386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:57.612755961 +0000 UTC m=+1801.407640456" watchObservedRunningTime="2026-10-30 11:59:58.048207386 +0000 UTC m=+1801.843091881"
Oct 30 11:59:58.049009 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.048991    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"]
Oct 30 11:59:58.055429 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.055408    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.075170 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.075144    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4093c5-dockercfg-dr2bc\""
Oct 30 11:59:58.085476 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.080187    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4093c5\""
Oct 30 11:59:58.144473 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.120724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:58.174531 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.145984    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2km\" (UniqueName: \"kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km\") pod \"payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.208489 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.178747    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.224486 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.189113    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"]
Oct 30 11:59:58.274815 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.274786    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"]
Oct 30 11:59:58.321654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.320789    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"]
Oct 30 11:59:58.321654 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.320897    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.355505 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.355122    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.356240 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.356219    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.356296 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.356280    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb4x\" (UniqueName: \"kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x\") pod \"payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.356375 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.356358    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.356704 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.356686    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2km\" (UniqueName: \"kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km\") pod \"payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.422480 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.421901    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2km\" (UniqueName: \"kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km\") pod \"payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:58.458168 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.458139    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb4x\" (UniqueName: \"kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x\") pod \"payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.458330 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.458176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.458891 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.458874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.502601 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.502567    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb4x\" (UniqueName: \"kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x\") pod \"payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.641022 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.640989    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:58.669591 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:58.669567    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:59.014562 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.004237    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"]
Oct 30 11:59:59.017626 ip-10-0-142-105 kubenswrapper[5060]: W1030 11:59:59.017516    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d047d9e_ebb3_4d19_97ed_0982641356e0.slice/crio-4e8f39f2d541d66f58684a41b3a4baf255e3b068de3398e63a2e39cb8d57f338 WatchSource:0}: Error finding container 4e8f39f2d541d66f58684a41b3a4baf255e3b068de3398e63a2e39cb8d57f338: Status 404 returned error can't find the container with id 4e8f39f2d541d66f58684a41b3a4baf255e3b068de3398e63a2e39cb8d57f338
Oct 30 11:59:59.123688 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.123665    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"]
Oct 30 11:59:59.174915 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.174843    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 11:59:59.612477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.605370    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf" event={"ID":"1d047d9e-ebb3-4d19-97ed-0982641356e0","Type":"ContainerStarted","Data":"42a38d7a62df913f9fae598982cefa839fee609946dcadb4a3cc653e207945fe"}
Oct 30 11:59:59.612477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.605438    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf" event={"ID":"1d047d9e-ebb3-4d19-97ed-0982641356e0","Type":"ContainerStarted","Data":"4e8f39f2d541d66f58684a41b3a4baf255e3b068de3398e63a2e39cb8d57f338"}
Oct 30 11:59:59.612477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.606414    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 11:59:59.612477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.608498    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q" event={"ID":"742cb5a2-d059-4fe4-a742-2e84248366d9","Type":"ContainerStarted","Data":"882db9eb24bc62d653c7e95882ef4fa0fffe7cf97d45dff3d837fe96ec80c539"}
Oct 30 11:59:59.612477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.608560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q" event={"ID":"742cb5a2-d059-4fe4-a742-2e84248366d9","Type":"ContainerStarted","Data":"cc84855721891fc5506bb0a5b40a4b2c10d48a045aeeec4a3c2734411ebc07f3"}
Oct 30 11:59:59.612477 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.609208    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 11:59:59.651519 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.651463    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf" podStartSLOduration=1.6514333319999999 podStartE2EDuration="1.651433332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:59.64970548 +0000 UTC m=+1803.444589974" watchObservedRunningTime="2026-10-30 11:59:59.651433332 +0000 UTC m=+1803.446317826"
Oct 30 11:59:59.697221 ip-10-0-142-105 kubenswrapper[5060]: I1030 11:59:59.697155    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q" podStartSLOduration=1.6971363560000001 podStartE2EDuration="1.697136356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:59.695849509 +0000 UTC m=+1803.490734000" watchObservedRunningTime="2026-10-30 11:59:59.697136356 +0000 UTC m=+1803.492020851"
Oct 30 12:00:00.146941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.146902    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"]
Oct 30 12:00:00.151433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.151408    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"]
Oct 30 12:00:00.151586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.151539    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.155368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.155344    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"]
Oct 30 12:00:00.155549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.155474    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.159530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.159504    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.159656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.159512    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"]
Oct 30 12:00:00.163796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.163778    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.173672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.173645    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.184169    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.184285    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h246t\" (UniqueName: \"kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.184317    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw4t\" (UniqueName: \"kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.184358    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splcd\" (UniqueName: \"kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.184480    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.184505    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.186604    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:00:00.195477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.187255    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:00:00.227892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.213119    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"]
Oct 30 12:00:00.244680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.244584    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"]
Oct 30 12:00:00.248794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.248767    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"]
Oct 30 12:00:00.257643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.256259    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"]
Oct 30 12:00:00.257643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.256354    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.258208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.258170    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"]
Oct 30 12:00:00.262126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.262068    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"]
Oct 30 12:00:00.272602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.266815    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.272602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.268535    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"]
Oct 30 12:00:00.274369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.273270    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"]
Oct 30 12:00:00.274369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.273379    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.276292    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"]
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302613    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsj7\" (UniqueName: \"kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302716    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h246t\" (UniqueName: \"kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302753    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw4t\" (UniqueName: \"kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302796    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splcd\" (UniqueName: \"kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcsv\" (UniqueName: \"kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv\") pod \"maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302860    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.302989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.303017    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.303133    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.303964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.308398    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.314592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.309054    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.328172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.328143    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"]
Oct 30 12:00:00.335080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.335043    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h246t\" (UniqueName: \"kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.347022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.346991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-splcd\" (UniqueName: \"kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.347216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.347189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw4t\" (UniqueName: \"kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.412602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.412436    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsj7\" (UniqueName: \"kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.413010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.412987    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcsv\" (UniqueName: \"kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv\") pod \"maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.413165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.413149    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.413415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.413374    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bm6\" (UniqueName: \"kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6\") pod \"maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.413545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.413434    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.413545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.413498    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.413545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.413519    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889360-f8svf\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.413679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.413578    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shh29\" (UniqueName: \"kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29\") pod \"maas-api-key-cleanup-29889360-f8svf\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.414325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.414304    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.414424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.414325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.436581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.436540    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsj7\" (UniqueName: \"kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.437808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.437777    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcsv\" (UniqueName: \"kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv\") pod \"maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.467762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.467733    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:00.515584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.515103    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bm6\" (UniqueName: \"kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6\") pod \"maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.515584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.515183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.515584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.515216    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889360-f8svf\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.515584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.515287    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shh29\" (UniqueName: \"kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29\") pod \"maas-api-key-cleanup-29889360-f8svf\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.517576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.517299    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.518628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.518219    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889360-f8svf\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.540528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.540496    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:00.540953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.540933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bm6\" (UniqueName: \"kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6\") pod \"maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.541567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.541546    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shh29\" (UniqueName: \"kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29\") pod \"maas-api-key-cleanup-29889360-f8svf\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.556899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.556872    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:00.560935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.560751    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:00.575564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.575364    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:00.652429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.652057    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:00.672256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.670385    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:00.835110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:00.835054    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"]
Oct 30 12:00:01.017478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.017233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:01.074946 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:01.074904    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f1eff3_c985_438a_9ea1_53abb8ff5833.slice/crio-7b71f6f4e5037245433af66dd4e314347ab5e32ed416ccaadd83bba6790e4a34 WatchSource:0}: Error finding container 7b71f6f4e5037245433af66dd4e314347ab5e32ed416ccaadd83bba6790e4a34: Status 404 returned error can't find the container with id 7b71f6f4e5037245433af66dd4e314347ab5e32ed416ccaadd83bba6790e4a34
Oct 30 12:00:01.075668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.075641    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"]
Oct 30 12:00:01.126818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.126205    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"]
Oct 30 12:00:01.135499 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:01.135416    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01420f8_1dde_455c_bad7_ddf9a9e5bbd1.slice/crio-0d14a6b6f596242354cc09b248bcef7928f2ee01648f7259396fdad8998d78e2 WatchSource:0}: Error finding container 0d14a6b6f596242354cc09b248bcef7928f2ee01648f7259396fdad8998d78e2: Status 404 returned error can't find the container with id 0d14a6b6f596242354cc09b248bcef7928f2ee01648f7259396fdad8998d78e2
Oct 30 12:00:01.163212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.163169    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:01.223740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.222428    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"]
Oct 30 12:00:01.227370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.227306    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"]
Oct 30 12:00:01.237226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.237187    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"]
Oct 30 12:00:01.323549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.323392    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"]
Oct 30 12:00:01.372814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.372756    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"]
Oct 30 12:00:01.639023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.638987    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerStarted","Data":"0031b587255877703e59981f4f93f253196e2d4f69658f3c9d92452241f72e6d"}
Oct 30 12:00:01.640337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.640315    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerStarted","Data":"dc27d8c65b7be2d22abe8a5ee789218952bbf440fe9db04fb165fa23b68e9858"}
Oct 30 12:00:01.641502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.641478    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerStarted","Data":"a8845eede07591e66fd5cc01abd8f55324aee3ebb1a655e586977dbd8ebac1ff"}
Oct 30 12:00:01.642554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.642532    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerStarted","Data":"9854f965c975948e2e28a2e98821fa4df2441e9108bbb79e517b719950106a34"}
Oct 30 12:00:01.644052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.644032    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerStarted","Data":"ae028ed06b3a942d99053602e52fd4570c7090e741fa28d814c36cfd12546058"}
Oct 30 12:00:01.644148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.644066    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerStarted","Data":"7b71f6f4e5037245433af66dd4e314347ab5e32ed416ccaadd83bba6790e4a34"}
Oct 30 12:00:01.646294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.646271    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerStarted","Data":"ed64f8935e237e02fd7e5ead02a3a022c1e7e75dcc505c77f7f90b6c3aadbdd6"}
Oct 30 12:00:01.647524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.647474    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerStarted","Data":"0d14a6b6f596242354cc09b248bcef7928f2ee01648f7259396fdad8998d78e2"}
Oct 30 12:00:01.696156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:01.696080    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" podStartSLOduration=1.696057813 podStartE2EDuration="1.696057813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:01.695817001 +0000 UTC m=+1805.490701497" watchObservedRunningTime="2026-10-30 12:00:01.696057813 +0000 UTC m=+1805.490942308"
Oct 30 12:00:02.659127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.659089    5060 generic.go:356] "Generic (PLEG): container finished" podID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerID="ae028ed06b3a942d99053602e52fd4570c7090e741fa28d814c36cfd12546058" exitCode=60
Oct 30 12:00:02.659645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.659171    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerDied","Data":"ae028ed06b3a942d99053602e52fd4570c7090e741fa28d814c36cfd12546058"}
Oct 30 12:00:02.659645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.659435    5060 scope.go:122] "RemoveContainer" containerID="ae028ed06b3a942d99053602e52fd4570c7090e741fa28d814c36cfd12546058"
Oct 30 12:00:02.661515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.661491    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerStarted","Data":"a8aa6dbc61c05cf4b7999f1b21eb8df9a4e1007f30b247682d69c3773746f85f"}
Oct 30 12:00:02.663375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.663349    5060 generic.go:356] "Generic (PLEG): container finished" podID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerID="079948d2c3152c0eb44e537a28cf555724b56dcdbc835e7a27872d9e81c426fd" exitCode=60
Oct 30 12:00:02.663512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.663428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerDied","Data":"079948d2c3152c0eb44e537a28cf555724b56dcdbc835e7a27872d9e81c426fd"}
Oct 30 12:00:02.663681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.663659    5060 scope.go:122] "RemoveContainer" containerID="079948d2c3152c0eb44e537a28cf555724b56dcdbc835e7a27872d9e81c426fd"
Oct 30 12:00:02.666681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.666659    5060 generic.go:356] "Generic (PLEG): container finished" podID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerID="579250464ea0e99496d15794445ff1978b54c1170c922af5790ea6d93cef2c01" exitCode=60
Oct 30 12:00:02.666794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.666731    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerDied","Data":"579250464ea0e99496d15794445ff1978b54c1170c922af5790ea6d93cef2c01"}
Oct 30 12:00:02.666974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.666959    5060 scope.go:122] "RemoveContainer" containerID="579250464ea0e99496d15794445ff1978b54c1170c922af5790ea6d93cef2c01"
Oct 30 12:00:02.668907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.668886    5060 generic.go:356] "Generic (PLEG): container finished" podID="55082e7a-26ec-4444-a1e6-83456c739776" containerID="6133420f5f7b00e498cedc1aa4d0644a1460afe456ca5db42dc5ed79fb96d232" exitCode=60
Oct 30 12:00:02.669060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.669041    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerDied","Data":"6133420f5f7b00e498cedc1aa4d0644a1460afe456ca5db42dc5ed79fb96d232"}
Oct 30 12:00:02.669473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.669122    5060 scope.go:122] "RemoveContainer" containerID="6133420f5f7b00e498cedc1aa4d0644a1460afe456ca5db42dc5ed79fb96d232"
Oct 30 12:00:02.671618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.671546    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerStarted","Data":"039f53c9a13bd7b93a7fe943f15593ce812288e11bb83bcfc493bdd9c47cceaa"}
Oct 30 12:00:02.674110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.674088    5060 generic.go:356] "Generic (PLEG): container finished" podID="c719e029-9961-4b8f-99cf-53bad8285414" containerID="f8b4e040897d47af941baf6146b39b31ea22e123ad41b56d1efcdd2f364d5e71" exitCode=60
Oct 30 12:00:02.674186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.674149    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerDied","Data":"f8b4e040897d47af941baf6146b39b31ea22e123ad41b56d1efcdd2f364d5e71"}
Oct 30 12:00:02.674395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.674376    5060 scope.go:122] "RemoveContainer" containerID="f8b4e040897d47af941baf6146b39b31ea22e123ad41b56d1efcdd2f364d5e71"
Oct 30 12:00:02.929334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.929278    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" podStartSLOduration=2.929262736 podStartE2EDuration="2.929262736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:02.925891174 +0000 UTC m=+1806.720775667" watchObservedRunningTime="2026-10-30 12:00:02.929262736 +0000 UTC m=+1806.724147247"
Oct 30 12:00:02.972516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.969414    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" podStartSLOduration=2.96939588 podStartE2EDuration="2.96939588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:02.966590735 +0000 UTC m=+1806.761475243" watchObservedRunningTime="2026-10-30 12:00:02.96939588 +0000 UTC m=+1806.764280393"
Oct 30 12:00:02.973348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:02.973302    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:03.689285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.689245    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerID="039f53c9a13bd7b93a7fe943f15593ce812288e11bb83bcfc493bdd9c47cceaa" exitCode=60
Oct 30 12:00:03.689815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.689334    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerDied","Data":"039f53c9a13bd7b93a7fe943f15593ce812288e11bb83bcfc493bdd9c47cceaa"}
Oct 30 12:00:03.689815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.689678    5060 scope.go:122] "RemoveContainer" containerID="039f53c9a13bd7b93a7fe943f15593ce812288e11bb83bcfc493bdd9c47cceaa"
Oct 30 12:00:03.692196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.692169    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerStarted","Data":"a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859"}
Oct 30 12:00:03.696713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.694285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerStarted","Data":"3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a"}
Oct 30 12:00:03.697498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.697468    5060 generic.go:356] "Generic (PLEG): container finished" podID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerID="a8aa6dbc61c05cf4b7999f1b21eb8df9a4e1007f30b247682d69c3773746f85f" exitCode=60
Oct 30 12:00:03.697602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.697548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerDied","Data":"a8aa6dbc61c05cf4b7999f1b21eb8df9a4e1007f30b247682d69c3773746f85f"}
Oct 30 12:00:03.697781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.697765    5060 scope.go:122] "RemoveContainer" containerID="a8aa6dbc61c05cf4b7999f1b21eb8df9a4e1007f30b247682d69c3773746f85f"
Oct 30 12:00:03.700022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.699998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerStarted","Data":"582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23"}
Oct 30 12:00:03.702435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.702410    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerStarted","Data":"dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1"}
Oct 30 12:00:03.716691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.716651    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerStarted","Data":"f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3"}
Oct 30 12:00:03.870697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.870642    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" podStartSLOduration=3.8706283729999997 podStartE2EDuration="3.870628373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:03.867100052 +0000 UTC m=+1807.661984545" watchObservedRunningTime="2026-10-30 12:00:03.870628373 +0000 UTC m=+1807.665512866"
Oct 30 12:00:03.962712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.962649    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" podStartSLOduration=3.962631649 podStartE2EDuration="3.962631649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:03.959992956 +0000 UTC m=+1807.754877451" watchObservedRunningTime="2026-10-30 12:00:03.962631649 +0000 UTC m=+1807.757516142"
Oct 30 12:00:03.963883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:03.963832    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" podStartSLOduration=3.963820973 podStartE2EDuration="3.963820973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:03.918696814 +0000 UTC m=+1807.713581309" watchObservedRunningTime="2026-10-30 12:00:03.963820973 +0000 UTC m=+1807.758705468"
Oct 30 12:00:04.003498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.003415    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" podStartSLOduration=4.003397803 podStartE2EDuration="4.003397803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:04.002642602 +0000 UTC m=+1807.797527096" watchObservedRunningTime="2026-10-30 12:00:04.003397803 +0000 UTC m=+1807.798282300"
Oct 30 12:00:04.727440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.727382    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerStarted","Data":"454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4"}
Oct 30 12:00:04.729539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.729508    5060 generic.go:356] "Generic (PLEG): container finished" podID="c719e029-9961-4b8f-99cf-53bad8285414" containerID="a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859" exitCode=60
Oct 30 12:00:04.729676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.729591    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerDied","Data":"a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859"}
Oct 30 12:00:04.729676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.729645    5060 scope.go:122] "RemoveContainer" containerID="f8b4e040897d47af941baf6146b39b31ea22e123ad41b56d1efcdd2f364d5e71"
Oct 30 12:00:04.729896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.729879    5060 scope.go:122] "RemoveContainer" containerID="a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859"
Oct 30 12:00:04.730255 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:04.730225    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq_odh-ai-gateway-infra(c719e029-9961-4b8f-99cf-53bad8285414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" podUID="c719e029-9961-4b8f-99cf-53bad8285414"
Oct 30 12:00:04.732078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.732048    5060 generic.go:356] "Generic (PLEG): container finished" podID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerID="3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a" exitCode=60
Oct 30 12:00:04.732181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.732106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerDied","Data":"3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a"}
Oct 30 12:00:04.732417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.732396    5060 scope.go:122] "RemoveContainer" containerID="3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a"
Oct 30 12:00:04.732765 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:04.732738    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf_odh-ai-gateway-infra(f2f1eff3-c985-438a-9ea1-53abb8ff5833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833"
Oct 30 12:00:04.734506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.734480    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerStarted","Data":"8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195"}
Oct 30 12:00:04.736629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.736598    5060 generic.go:356] "Generic (PLEG): container finished" podID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerID="582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23" exitCode=60
Oct 30 12:00:04.736734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.736632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerDied","Data":"582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23"}
Oct 30 12:00:04.736878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.736863    5060 scope.go:122] "RemoveContainer" containerID="582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23"
Oct 30 12:00:04.737086 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:04.737065    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms_odh-ai-gateway-infra(e01420f8-1dde-455c-bad7-ddf9a9e5bbd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1"
Oct 30 12:00:04.738938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.738824    5060 generic.go:356] "Generic (PLEG): container finished" podID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerID="dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1" exitCode=60
Oct 30 12:00:04.738938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.738852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerDied","Data":"dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1"}
Oct 30 12:00:04.739210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.739183    5060 scope.go:122] "RemoveContainer" containerID="dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1"
Oct 30 12:00:04.739509 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:04.739440    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl_odh-ai-gateway-infra(26ca9220-c784-42ec-9d4d-c5dcf240dae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1"
Oct 30 12:00:04.741575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.741553    5060 generic.go:356] "Generic (PLEG): container finished" podID="55082e7a-26ec-4444-a1e6-83456c739776" containerID="f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3" exitCode=60
Oct 30 12:00:04.741668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.741631    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerDied","Data":"f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3"}
Oct 30 12:00:04.741797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.741784    5060 scope.go:122] "RemoveContainer" containerID="f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3"
Oct 30 12:00:04.741964 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:04.741949    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2_odh-ai-gateway-infra(55082e7a-26ec-4444-a1e6-83456c739776)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" podUID="55082e7a-26ec-4444-a1e6-83456c739776"
Oct 30 12:00:04.747350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.747335    5060 scope.go:122] "RemoveContainer" containerID="ae028ed06b3a942d99053602e52fd4570c7090e741fa28d814c36cfd12546058"
Oct 30 12:00:04.755931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.755898    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:04.766773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.765708    5060 scope.go:122] "RemoveContainer" containerID="079948d2c3152c0eb44e537a28cf555724b56dcdbc835e7a27872d9e81c426fd"
Oct 30 12:00:04.798932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.798899    5060 scope.go:122] "RemoveContainer" containerID="579250464ea0e99496d15794445ff1978b54c1170c922af5790ea6d93cef2c01"
Oct 30 12:00:04.815782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:04.815653    5060 scope.go:122] "RemoveContainer" containerID="6133420f5f7b00e498cedc1aa4d0644a1460afe456ca5db42dc5ed79fb96d232"
Oct 30 12:00:05.755033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.754997    5060 scope.go:122] "RemoveContainer" containerID="3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a"
Oct 30 12:00:05.755416 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.755356    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf_odh-ai-gateway-infra(f2f1eff3-c985-438a-9ea1-53abb8ff5833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833"
Oct 30 12:00:05.758159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.758130    5060 generic.go:356] "Generic (PLEG): container finished" podID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerID="8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195" exitCode=60
Oct 30 12:00:05.758301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.758201    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerDied","Data":"8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195"}
Oct 30 12:00:05.758301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.758269    5060 scope.go:122] "RemoveContainer" containerID="a8aa6dbc61c05cf4b7999f1b21eb8df9a4e1007f30b247682d69c3773746f85f"
Oct 30 12:00:05.758516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.758497    5060 scope.go:122] "RemoveContainer" containerID="8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195"
Oct 30 12:00:05.758781 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.758763    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8_odh-ai-gateway-infra(e2faa273-0952-4c66-a8f5-adb99e72580c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c"
Oct 30 12:00:05.760750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.760733    5060 scope.go:122] "RemoveContainer" containerID="582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23"
Oct 30 12:00:05.760996 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.760975    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms_odh-ai-gateway-infra(e01420f8-1dde-455c-bad7-ddf9a9e5bbd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1"
Oct 30 12:00:05.762556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.762538    5060 scope.go:122] "RemoveContainer" containerID="dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1"
Oct 30 12:00:05.762785 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.762762    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl_odh-ai-gateway-infra(26ca9220-c784-42ec-9d4d-c5dcf240dae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1"
Oct 30 12:00:05.764334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.764316    5060 scope.go:122] "RemoveContainer" containerID="f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3"
Oct 30 12:00:05.764570 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.764549    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2_odh-ai-gateway-infra(55082e7a-26ec-4444-a1e6-83456c739776)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" podUID="55082e7a-26ec-4444-a1e6-83456c739776"
Oct 30 12:00:05.766172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.766148    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerID="454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4" exitCode=60
Oct 30 12:00:05.766265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.766221    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerDied","Data":"454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4"}
Oct 30 12:00:05.766433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.766416    5060 scope.go:122] "RemoveContainer" containerID="454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4"
Oct 30 12:00:05.766718 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.766669    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889360-f8svf_odh-ai-gateway-infra(0ccda7bc-8e19-4b5b-a87d-7409c195d182)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182"
Oct 30 12:00:05.768348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.768329    5060 scope.go:122] "RemoveContainer" containerID="a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859"
Oct 30 12:00:05.768573 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:05.768554    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq_odh-ai-gateway-infra(c719e029-9961-4b8f-99cf-53bad8285414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" podUID="c719e029-9961-4b8f-99cf-53bad8285414"
Oct 30 12:00:05.772948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.772927    5060 scope.go:122] "RemoveContainer" containerID="039f53c9a13bd7b93a7fe943f15593ce812288e11bb83bcfc493bdd9c47cceaa"
Oct 30 12:00:05.836244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:05.836204    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm" podUID="26d0a6ed-db94-4ce0-9bad-6be745ff99e1" containerName="payload-pre-processing" containerID="cri-o://dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b" gracePeriod=25
Oct 30 12:00:06.152113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.152084    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 12:00:06.200711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.200676    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn\" (UniqueName: \"kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn\") pod \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") "
Oct 30 12:00:06.200910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.200824    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume\") pod \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\" (UID: \"26d0a6ed-db94-4ce0-9bad-6be745ff99e1\") "
Oct 30 12:00:06.201182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.201159    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume" pod "26d0a6ed-db94-4ce0-9bad-6be745ff99e1" (UID: "26d0a6ed-db94-4ce0-9bad-6be745ff99e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:06.204651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.204620    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn" pod "26d0a6ed-db94-4ce0-9bad-6be745ff99e1" (UID: "26d0a6ed-db94-4ce0-9bad-6be745ff99e1"). InnerVolumeSpecName "kube-api-access-5zdpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:06.303045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.302958    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdpn\" (UniqueName: \"kubernetes.io/projected/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-kube-api-access-5zdpn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:06.303045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.302994    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d0a6ed-db94-4ce0-9bad-6be745ff99e1-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:06.615621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.615528    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:06.778071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.778040    5060 scope.go:122] "RemoveContainer" containerID="8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195"
Oct 30 12:00:06.778591 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:06.778402    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8_odh-ai-gateway-infra(e2faa273-0952-4c66-a8f5-adb99e72580c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c"
Oct 30 12:00:06.779379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.779349    5060 generic.go:356] "Generic (PLEG): container finished" podID="26d0a6ed-db94-4ce0-9bad-6be745ff99e1" containerID="dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b" exitCode=0
Oct 30 12:00:06.779546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.779399    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"
Oct 30 12:00:06.779546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.779428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm" event={"ID":"26d0a6ed-db94-4ce0-9bad-6be745ff99e1","Type":"ContainerDied","Data":"dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b"}
Oct 30 12:00:06.779546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.779499    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm" event={"ID":"26d0a6ed-db94-4ce0-9bad-6be745ff99e1","Type":"ContainerDied","Data":"0c05e39d3dd464e26d7f34b18dc6ca964453f188e97facc442b1e698d226fd57"}
Oct 30 12:00:06.779546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.779523    5060 scope.go:122] "RemoveContainer" containerID="dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b"
Oct 30 12:00:06.782405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.781934    5060 scope.go:122] "RemoveContainer" containerID="454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4"
Oct 30 12:00:06.782405 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:06.782240    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889360-f8svf_odh-ai-gateway-infra(0ccda7bc-8e19-4b5b-a87d-7409c195d182)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182"
Oct 30 12:00:06.784215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.784195    5060 generic.go:356] "Generic (PLEG): container finished" podID="0952119e-66dd-4e14-95f0-f5971be5c762" containerID="71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70" exitCode=0
Oct 30 12:00:06.784313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.784228    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" event={"ID":"0952119e-66dd-4e14-95f0-f5971be5c762","Type":"ContainerDied","Data":"71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"}
Oct 30 12:00:06.784684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.784667    5060 scope.go:122] "RemoveContainer" containerID="71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"
Oct 30 12:00:06.786371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.786341    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:06.786481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.786425    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:06.786527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.786504    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:06.791409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.791371    5060 scope.go:122] "RemoveContainer" containerID="dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b"
Oct 30 12:00:06.791726 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:06.791698    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b\": container with ID starting with dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b not found: ID does not exist" containerID="dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b"
Oct 30 12:00:06.791818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.791736    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b"} err="failed to get container status \"dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b\": rpc error: code = NotFound desc = could not find container \"dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b\": container with ID starting with dd2348d44cf5a4670b62d58b68540b90011fb4fba60ece0c06d95b6553f4b84b not found: ID does not exist"
Oct 30 12:00:06.858392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.858358    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"]
Oct 30 12:00:06.864946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:06.864921    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6fcda5-9658fdf75-ctrhm"]
Oct 30 12:00:07.330268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.330231    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"]
Oct 30 12:00:07.330975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.330954    5060 cpu_manager.go:414] "containerMap: removing container" podUID="26d0a6ed-db94-4ce0-9bad-6be745ff99e1" containerName="payload-pre-processing"
Oct 30 12:00:07.330975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.330974    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d0a6ed-db94-4ce0-9bad-6be745ff99e1" containerName="payload-pre-processing"
Oct 30 12:00:07.331140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.331090    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d0a6ed-db94-4ce0-9bad-6be745ff99e1" containerName="payload-pre-processing"
Oct 30 12:00:07.334742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.334723    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.393503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.382381    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"]
Oct 30 12:00:07.436925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.436887    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.436925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.436943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z527k\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437220    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437250    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437349    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437393    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437518    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.437599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.437584    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538193    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538248    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538269    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538302    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z527k\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538371    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538491    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538566    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538753    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538762    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.538999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.538910    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.539282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.539260    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.539487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.539444    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.541059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.541036    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.541329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.541311    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.568792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.568759    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.571524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.571496    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z527k\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.682533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.679718    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:07.823493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.819727    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" event={"ID":"0952119e-66dd-4e14-95f0-f5971be5c762","Type":"ContainerStarted","Data":"77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902"}
Oct 30 12:00:07.961197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.961163    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 12:00:07.962557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.962533    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 12:00:07.962685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:07.962580    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 12:00:08.028927 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:08.028895    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc396b539_68c8_44d1_88e8_45117a4f7ea9.slice/crio-74d6c9f2a0af24610aaff6fe9848d07e25450cae9e14c01cd3b68bd78bd5df41 WatchSource:0}: Error finding container 74d6c9f2a0af24610aaff6fe9848d07e25450cae9e14c01cd3b68bd78bd5df41: Status 404 returned error can't find the container with id 74d6c9f2a0af24610aaff6fe9848d07e25450cae9e14c01cd3b68bd78bd5df41
Oct 30 12:00:08.030381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.030357    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"]
Oct 30 12:00:08.031657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.031630    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:08.031755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.031687    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:08.031755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.031716    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:08.337486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.337376    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d0a6ed-db94-4ce0-9bad-6be745ff99e1" path="/var/lib/kubelet/pods/26d0a6ed-db94-4ce0-9bad-6be745ff99e1/volumes"
Oct 30 12:00:08.608981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.607614    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 12:00:08.764752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.764720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:08.847341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.847310    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" event={"ID":"c396b539-68c8-44d1-88e8-45117a4f7ea9","Type":"ContainerStarted","Data":"fce1ab6a75f2197dfed3638e756ed3158b3e6840ac75870a72632bc264f28ccc"}
Oct 30 12:00:08.847885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.847365    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" event={"ID":"c396b539-68c8-44d1-88e8-45117a4f7ea9","Type":"ContainerStarted","Data":"74d6c9f2a0af24610aaff6fe9848d07e25450cae9e14c01cd3b68bd78bd5df41"}
Oct 30 12:00:08.961382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.961334    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 12:00:08.961579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:08.961397    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 12:00:09.138606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.138567    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:09.309538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.309403    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:09.683538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.682609    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:09.687474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.684474    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 12:00:09.687474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.684531    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 12:00:09.961309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.961275    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 12:00:09.961793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:09.961769    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 12:00:10.685477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:10.684881    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 12:00:10.685477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:10.684947    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 12:00:10.956872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:10.956778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:10.960754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:10.960728    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 12:00:10.960896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:10.960783    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 12:00:11.138505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.135598    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podStartSLOduration=4.135573905 podStartE2EDuration="4.135573905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:08.911808574 +0000 UTC m=+1812.706693070" watchObservedRunningTime="2026-10-30 12:00:11.135573905 +0000 UTC m=+1814.930458400"
Oct 30 12:00:11.139046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.138756    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"]
Oct 30 12:00:11.145182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.145157    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.163486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.163433    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"]
Oct 30 12:00:11.171301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.171272    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-eb5531-dockercfg-xsbxx\""
Oct 30 12:00:11.171494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.171371    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-eb5531\""
Oct 30 12:00:11.253533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.248941    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf28k\" (UniqueName: \"kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k\") pod \"payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.253533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.249093    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.291506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.284029    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:11.387480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.381562    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.387480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.381757    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf28k\" (UniqueName: \"kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k\") pod \"payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.413478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.413164    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.468721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.463728    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:11.511649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.511566    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf28k\" (UniqueName: \"kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k\") pod \"payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.651543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.651508    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 12:00:11.652304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.652282    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 12:00:11.691477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.685862    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 12:00:11.691477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.685922    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 12:00:11.764705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.764611    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:11.801613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.801574    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 12:00:11.961296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.960991    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 12:00:11.961296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:11.961049    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 12:00:12.235480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.235105    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"]
Oct 30 12:00:12.241803 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:12.241752    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a97f6f_bbc0_456c_a3bc_2b80c75ba406.slice/crio-656dfad0dce588c12df76235a181659b78a654c52e64d465c82046b259df5d4d WatchSource:0}: Error finding container 656dfad0dce588c12df76235a181659b78a654c52e64d465c82046b259df5d4d: Status 404 returned error can't find the container with id 656dfad0dce588c12df76235a181659b78a654c52e64d465c82046b259df5d4d
Oct 30 12:00:12.697540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.682956    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 12:00:12.697540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.683017    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 12:00:12.959427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.959297    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj" event={"ID":"17a97f6f-bbc0-456c-a3bc-2b80c75ba406","Type":"ContainerStarted","Data":"b9b053191770076807a5e0c9d227351de4f3fa6465ec84635701ed8a0adc35bb"}
Oct 30 12:00:12.959427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.959409    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj" event={"ID":"17a97f6f-bbc0-456c-a3bc-2b80c75ba406","Type":"ContainerStarted","Data":"656dfad0dce588c12df76235a181659b78a654c52e64d465c82046b259df5d4d"}
Oct 30 12:00:12.959678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.959637    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:12.960749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.960725    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 12:00:12.960858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:12.960767    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 12:00:13.312666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:13.312552    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:13.687536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:13.683871    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 12:00:13.687536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:13.683933    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 12:00:13.965041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:13.965012    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 12:00:13.966286 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:13.966263    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 12:00:14.055729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:14.055678    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj" podStartSLOduration=3.055662487 podStartE2EDuration="3.055662487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:13.009651971 +0000 UTC m=+1816.804536466" watchObservedRunningTime="2026-10-30 12:00:14.055662487 +0000 UTC m=+1817.850546980"
Oct 30 12:00:14.327868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:14.327784    5060 scope.go:122] "RemoveContainer" containerID="f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3"
Oct 30 12:00:14.687871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:14.687838    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:14.688220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:14.688200    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:14.689101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:14.689078    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:00:14.982354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:14.982320    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerStarted","Data":"f593a25f898022b471d7ea8e38a805cda620c0010ec56c04723701460b38de4f"}
Oct 30 12:00:15.297367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.297338    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:15.328030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.327989    5060 scope.go:122] "RemoveContainer" containerID="3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a"
Oct 30 12:00:15.328503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.328065    5060 scope.go:122] "RemoveContainer" containerID="a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859"
Oct 30 12:00:15.993971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.993929    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerStarted","Data":"f9408444bf5beeb38f94e19ce1365ca889182ac17fb744af5bdc1853aed28fa2"}
Oct 30 12:00:15.996265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.996234    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerStarted","Data":"2c2dec2e2db25b28b62f5d8c377feb39d3375078424abd50fa05df90a663dc92"}
Oct 30 12:00:15.998538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.998509    5060 generic.go:356] "Generic (PLEG): container finished" podID="55082e7a-26ec-4444-a1e6-83456c739776" containerID="f593a25f898022b471d7ea8e38a805cda620c0010ec56c04723701460b38de4f" exitCode=60
Oct 30 12:00:15.998664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.998582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerDied","Data":"f593a25f898022b471d7ea8e38a805cda620c0010ec56c04723701460b38de4f"}
Oct 30 12:00:15.998664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.998649    5060 scope.go:122] "RemoveContainer" containerID="f942d13a077dbb051e0fcfded7c093674b0c795ff92aadfdde0635f2c19214f3"
Oct 30 12:00:15.998990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:15.998974    5060 scope.go:122] "RemoveContainer" containerID="f593a25f898022b471d7ea8e38a805cda620c0010ec56c04723701460b38de4f"
Oct 30 12:00:15.999285 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:15.999261    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2_odh-ai-gateway-infra(55082e7a-26ec-4444-a1e6-83456c739776)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" podUID="55082e7a-26ec-4444-a1e6-83456c739776"
Oct 30 12:00:16.035348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:16.035319    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"]
Oct 30 12:00:16.337302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:16.337272    5060 scope.go:122] "RemoveContainer" containerID="582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23"
Oct 30 12:00:17.008849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.008818    5060 generic.go:356] "Generic (PLEG): container finished" podID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerID="2c2dec2e2db25b28b62f5d8c377feb39d3375078424abd50fa05df90a663dc92" exitCode=60
Oct 30 12:00:17.009051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.008898    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerDied","Data":"2c2dec2e2db25b28b62f5d8c377feb39d3375078424abd50fa05df90a663dc92"}
Oct 30 12:00:17.009051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.008957    5060 scope.go:122] "RemoveContainer" containerID="3cc4071a688ab9bf7d4f8e3aac4dd1ac8f2be3734d36627196db383c394dd95a"
Oct 30 12:00:17.009340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.009323    5060 scope.go:122] "RemoveContainer" containerID="2c2dec2e2db25b28b62f5d8c377feb39d3375078424abd50fa05df90a663dc92"
Oct 30 12:00:17.009682 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:17.009660    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf_odh-ai-gateway-infra(f2f1eff3-c985-438a-9ea1-53abb8ff5833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833"
Oct 30 12:00:17.011914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.011890    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerStarted","Data":"666a8402161d72e42e7904138afd160953a76072aa28c61b1cd21cd6f8ab0875"}
Oct 30 12:00:17.016977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.016950    5060 generic.go:356] "Generic (PLEG): container finished" podID="c719e029-9961-4b8f-99cf-53bad8285414" containerID="f9408444bf5beeb38f94e19ce1365ca889182ac17fb744af5bdc1853aed28fa2" exitCode=60
Oct 30 12:00:17.017070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.017018    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerDied","Data":"f9408444bf5beeb38f94e19ce1365ca889182ac17fb744af5bdc1853aed28fa2"}
Oct 30 12:00:17.017382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.017363    5060 scope.go:122] "RemoveContainer" containerID="f9408444bf5beeb38f94e19ce1365ca889182ac17fb744af5bdc1853aed28fa2"
Oct 30 12:00:17.017678 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:17.017657    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq_odh-ai-gateway-infra(c719e029-9961-4b8f-99cf-53bad8285414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" podUID="c719e029-9961-4b8f-99cf-53bad8285414"
Oct 30 12:00:17.025887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.025865    5060 scope.go:122] "RemoveContainer" containerID="a5de01c62501413b70da78964fcdabedd01a820cd43d2c842ad2a9ab6ff49859"
Oct 30 12:00:17.050526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.050494    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"]
Oct 30 12:00:17.131703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.131672    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"]
Oct 30 12:00:17.203991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.203959    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:17.214343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.214312    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7\" (UniqueName: \"kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7\") pod \"55082e7a-26ec-4444-a1e6-83456c739776\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") "
Oct 30 12:00:17.214556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.214394    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca\") pod \"55082e7a-26ec-4444-a1e6-83456c739776\" (UID: \"55082e7a-26ec-4444-a1e6-83456c739776\") "
Oct 30 12:00:17.214753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.214730    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca" pod "55082e7a-26ec-4444-a1e6-83456c739776" (UID: "55082e7a-26ec-4444-a1e6-83456c739776"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:17.214928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.214913    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55082e7a-26ec-4444-a1e6-83456c739776-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:17.216821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.216795    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7" pod "55082e7a-26ec-4444-a1e6-83456c739776" (UID: "55082e7a-26ec-4444-a1e6-83456c739776"). InnerVolumeSpecName "kube-api-access-xfsj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:17.316513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.316475    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfsj7\" (UniqueName: \"kubernetes.io/projected/55082e7a-26ec-4444-a1e6-83456c739776-kube-api-access-xfsj7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:17.635627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.635534    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:17.783844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:17.783810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:18.030244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.030209    5060 generic.go:356] "Generic (PLEG): container finished" podID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerID="666a8402161d72e42e7904138afd160953a76072aa28c61b1cd21cd6f8ab0875" exitCode=60
Oct 30 12:00:18.030348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.030274    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerDied","Data":"666a8402161d72e42e7904138afd160953a76072aa28c61b1cd21cd6f8ab0875"}
Oct 30 12:00:18.030348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.030333    5060 scope.go:122] "RemoveContainer" containerID="582af800088b0d11fa8d510fefc4393a58bee2968810be56befb16a557d8ec23"
Oct 30 12:00:18.030771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.030740    5060 scope.go:122] "RemoveContainer" containerID="666a8402161d72e42e7904138afd160953a76072aa28c61b1cd21cd6f8ab0875"
Oct 30 12:00:18.031094 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:18.031069    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms_odh-ai-gateway-infra(e01420f8-1dde-455c-bad7-ddf9a9e5bbd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1"
Oct 30 12:00:18.032112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.032086    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2" event={"ID":"55082e7a-26ec-4444-a1e6-83456c739776","Type":"ContainerDied","Data":"dc27d8c65b7be2d22abe8a5ee789218952bbf440fe9db04fb165fa23b68e9858"}
Oct 30 12:00:18.032293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.032277    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"
Oct 30 12:00:18.052949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.052922    5060 scope.go:122] "RemoveContainer" containerID="f593a25f898022b471d7ea8e38a805cda620c0010ec56c04723701460b38de4f"
Oct 30 12:00:18.154381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.154346    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"]
Oct 30 12:00:18.174571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.174540    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"]
Oct 30 12:00:18.187578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.187528    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889360-z84p2"]
Oct 30 12:00:18.243600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.243570    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:18.279313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.279235    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:18.328477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.328411    5060 scope.go:122] "RemoveContainer" containerID="8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195"
Oct 30 12:00:18.331996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.330944    5060 scope.go:122] "RemoveContainer" containerID="454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4"
Oct 30 12:00:18.331996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.331762    5060 scope.go:122] "RemoveContainer" containerID="dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1"
Oct 30 12:00:18.337347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.337065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t\" (UniqueName: \"kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t\") pod \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") "
Oct 30 12:00:18.337347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.337243    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca\") pod \"c719e029-9961-4b8f-99cf-53bad8285414\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") "
Oct 30 12:00:18.337508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.337356    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6\" (UniqueName: \"kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6\") pod \"c719e029-9961-4b8f-99cf-53bad8285414\" (UID: \"c719e029-9961-4b8f-99cf-53bad8285414\") "
Oct 30 12:00:18.337508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.337418    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca\") pod \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\" (UID: \"f2f1eff3-c985-438a-9ea1-53abb8ff5833\") "
Oct 30 12:00:18.338602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.337932    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca" pod "c719e029-9961-4b8f-99cf-53bad8285414" (UID: "c719e029-9961-4b8f-99cf-53bad8285414"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:18.338602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.338257    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca" pod "f2f1eff3-c985-438a-9ea1-53abb8ff5833" (UID: "f2f1eff3-c985-438a-9ea1-53abb8ff5833"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:18.350191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.350149    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t" pod "f2f1eff3-c985-438a-9ea1-53abb8ff5833" (UID: "f2f1eff3-c985-438a-9ea1-53abb8ff5833"). InnerVolumeSpecName "kube-api-access-rzw4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:18.351032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.350525    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6" pod "c719e029-9961-4b8f-99cf-53bad8285414" (UID: "c719e029-9961-4b8f-99cf-53bad8285414"). InnerVolumeSpecName "kube-api-access-d2bm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:18.403312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.377928    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55082e7a-26ec-4444-a1e6-83456c739776" path="/var/lib/kubelet/pods/55082e7a-26ec-4444-a1e6-83456c739776/volumes"
Oct 30 12:00:18.442473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.440119    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bm6\" (UniqueName: \"kubernetes.io/projected/c719e029-9961-4b8f-99cf-53bad8285414-kube-api-access-d2bm6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:18.442473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.440185    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f1eff3-c985-438a-9ea1-53abb8ff5833-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:18.442473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.440201    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzw4t\" (UniqueName: \"kubernetes.io/projected/f2f1eff3-c985-438a-9ea1-53abb8ff5833-kube-api-access-rzw4t\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:18.442473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:18.440221    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719e029-9961-4b8f-99cf-53bad8285414-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:19.051546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.051505    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf" event={"ID":"f2f1eff3-c985-438a-9ea1-53abb8ff5833","Type":"ContainerDied","Data":"7b71f6f4e5037245433af66dd4e314347ab5e32ed416ccaadd83bba6790e4a34"}
Oct 30 12:00:19.051546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.051542    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"
Oct 30 12:00:19.052071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.051575    5060 scope.go:122] "RemoveContainer" containerID="2c2dec2e2db25b28b62f5d8c377feb39d3375078424abd50fa05df90a663dc92"
Oct 30 12:00:19.054377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.054342    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerStarted","Data":"f9ac2beb787cd210df82e0f2259df1e2582bad6eca22465f5f65f302dd169112"}
Oct 30 12:00:19.058654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.058626    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerStarted","Data":"b74ec41e32b824d493d077c13f2e11207556d24eaf0b443b369d382404e2ae56"}
Oct 30 12:00:19.062252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.062224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerStarted","Data":"038ffa44ffcf890798b5012744cb95396f8606c1ae236eecfc63a6f28414e5d3"}
Oct 30 12:00:19.063762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.063732    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq" event={"ID":"c719e029-9961-4b8f-99cf-53bad8285414","Type":"ContainerDied","Data":"9854f965c975948e2e28a2e98821fa4df2441e9108bbb79e517b719950106a34"}
Oct 30 12:00:19.063868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.063825    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"
Oct 30 12:00:19.064822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.064782    5060 scope.go:122] "RemoveContainer" containerID="f9408444bf5beeb38f94e19ce1365ca889182ac17fb744af5bdc1853aed28fa2"
Oct 30 12:00:19.148800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.148764    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"]
Oct 30 12:00:19.151092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.151066    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6fcda5-29889360-bmnjq"]
Oct 30 12:00:19.262803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.260335    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:19.292218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.292038    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"]
Oct 30 12:00:19.301112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.301074    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889360-9szdf"]
Oct 30 12:00:19.348327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.348228    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:19.351913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.351883    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd\" (UniqueName: \"kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd\") pod \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") "
Oct 30 12:00:19.352136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.352116    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca\") pod \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\" (UID: \"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1\") "
Oct 30 12:00:19.352640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.352529    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca" pod "e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" (UID: "e01420f8-1dde-455c-bad7-ddf9a9e5bbd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:19.352818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.352677    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:19.354378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.354350    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd" pod "e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" (UID: "e01420f8-1dde-455c-bad7-ddf9a9e5bbd1"). InnerVolumeSpecName "kube-api-access-splcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:19.454233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.454183    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splcd\" (UniqueName: \"kubernetes.io/projected/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1-kube-api-access-splcd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:19.857741 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:19.857705    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:00:19.964422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:19.964386    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:20.073801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.073773    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"
Oct 30 12:00:20.074304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.073805    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms" event={"ID":"e01420f8-1dde-455c-bad7-ddf9a9e5bbd1","Type":"ContainerDied","Data":"0d14a6b6f596242354cc09b248bcef7928f2ee01648f7259396fdad8998d78e2"}
Oct 30 12:00:20.074304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.073877    5060 scope.go:122] "RemoveContainer" containerID="666a8402161d72e42e7904138afd160953a76072aa28c61b1cd21cd6f8ab0875"
Oct 30 12:00:20.077280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.077251    5060 generic.go:356] "Generic (PLEG): container finished" podID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerID="b74ec41e32b824d493d077c13f2e11207556d24eaf0b443b369d382404e2ae56" exitCode=60
Oct 30 12:00:20.077419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.077320    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerDied","Data":"b74ec41e32b824d493d077c13f2e11207556d24eaf0b443b369d382404e2ae56"}
Oct 30 12:00:20.077813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.077794    5060 scope.go:122] "RemoveContainer" containerID="b74ec41e32b824d493d077c13f2e11207556d24eaf0b443b369d382404e2ae56"
Oct 30 12:00:20.078161 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:20.078134    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl_odh-ai-gateway-infra(26ca9220-c784-42ec-9d4d-c5dcf240dae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1"
Oct 30 12:00:20.080055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.080023    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerID="038ffa44ffcf890798b5012744cb95396f8606c1ae236eecfc63a6f28414e5d3" exitCode=60
Oct 30 12:00:20.080166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.080095    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerDied","Data":"038ffa44ffcf890798b5012744cb95396f8606c1ae236eecfc63a6f28414e5d3"}
Oct 30 12:00:20.080542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.080524    5060 scope.go:122] "RemoveContainer" containerID="038ffa44ffcf890798b5012744cb95396f8606c1ae236eecfc63a6f28414e5d3"
Oct 30 12:00:20.080881 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:20.080858    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889360-f8svf_odh-ai-gateway-infra(0ccda7bc-8e19-4b5b-a87d-7409c195d182)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182"
Oct 30 12:00:20.087874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.087849    5060 generic.go:356] "Generic (PLEG): container finished" podID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerID="f9ac2beb787cd210df82e0f2259df1e2582bad6eca22465f5f65f302dd169112" exitCode=60
Oct 30 12:00:20.088002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.087896    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerDied","Data":"f9ac2beb787cd210df82e0f2259df1e2582bad6eca22465f5f65f302dd169112"}
Oct 30 12:00:20.088237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.088224    5060 scope.go:122] "RemoveContainer" containerID="f9ac2beb787cd210df82e0f2259df1e2582bad6eca22465f5f65f302dd169112"
Oct 30 12:00:20.088537 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:20.088513    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8_odh-ai-gateway-infra(e2faa273-0952-4c66-a8f5-adb99e72580c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c"
Oct 30 12:00:20.096198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.096174    5060 scope.go:122] "RemoveContainer" containerID="dd3fe15aa7e42a863eef59c8e02a62266b1dbf798f4cf104ce14cb37f31a07c1"
Oct 30 12:00:20.111431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.111412    5060 scope.go:122] "RemoveContainer" containerID="454683139cada1556e2ff0ff5b4a4aba550876d785a27b32c1e269d50c19e2b4"
Oct 30 12:00:20.118966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.118936    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"]
Oct 30 12:00:20.122964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.122938    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"]
Oct 30 12:00:20.126721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.126693    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6fcda5-9qtv5"]
Oct 30 12:00:20.127390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.127374    5060 scope.go:122] "RemoveContainer" containerID="8a08228b3c347a78571803ddcbcde8109f4334c8f5e63b7c05d5b81ce08ff195"
Oct 30 12:00:20.129850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.129821    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:20.174657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.174623    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"]
Oct 30 12:00:20.178751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.178709    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889360-kvlms"]
Oct 30 12:00:20.210582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.210546    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"]
Oct 30 12:00:20.250205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.250122    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"]
Oct 30 12:00:20.274540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.274504    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:20.335404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.335375    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c719e029-9961-4b8f-99cf-53bad8285414" path="/var/lib/kubelet/pods/c719e029-9961-4b8f-99cf-53bad8285414/volumes"
Oct 30 12:00:20.335890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.335875    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01b5e2c-97c3-495b-b02f-5fc051b28aaa" path="/var/lib/kubelet/pods/d01b5e2c-97c3-495b-b02f-5fc051b28aaa/volumes"
Oct 30 12:00:20.336231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.336218    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" path="/var/lib/kubelet/pods/e01420f8-1dde-455c-bad7-ddf9a9e5bbd1/volumes"
Oct 30 12:00:20.336977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:20.336961    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" path="/var/lib/kubelet/pods/f2f1eff3-c985-438a-9ea1-53abb8ff5833/volumes"
Oct 30 12:00:21.399155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.399125    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:21.402637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.402597    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:21.407414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.407388    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:21.482431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.482381    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca\") pod \"e2faa273-0952-4c66-a8f5-adb99e72580c\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") "
Oct 30 12:00:21.482662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.482498    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca\") pod \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") "
Oct 30 12:00:21.482662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.482564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca\") pod \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") "
Oct 30 12:00:21.482823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.482689    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t\" (UniqueName: \"kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t\") pod \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\" (UID: \"26ca9220-c784-42ec-9d4d-c5dcf240dae1\") "
Oct 30 12:00:21.482823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.482792    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv\" (UniqueName: \"kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv\") pod \"e2faa273-0952-4c66-a8f5-adb99e72580c\" (UID: \"e2faa273-0952-4c66-a8f5-adb99e72580c\") "
Oct 30 12:00:21.482943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.482820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29\" (UniqueName: \"kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29\") pod \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\" (UID: \"0ccda7bc-8e19-4b5b-a87d-7409c195d182\") "
Oct 30 12:00:21.484483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.483439    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca" pod "26ca9220-c784-42ec-9d4d-c5dcf240dae1" (UID: "26ca9220-c784-42ec-9d4d-c5dcf240dae1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:21.484483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.483798    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca" pod "e2faa273-0952-4c66-a8f5-adb99e72580c" (UID: "e2faa273-0952-4c66-a8f5-adb99e72580c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:21.484483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.484133    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca" pod "0ccda7bc-8e19-4b5b-a87d-7409c195d182" (UID: "0ccda7bc-8e19-4b5b-a87d-7409c195d182"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:21.487303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.487264    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29" pod "0ccda7bc-8e19-4b5b-a87d-7409c195d182" (UID: "0ccda7bc-8e19-4b5b-a87d-7409c195d182"). InnerVolumeSpecName "kube-api-access-shh29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:21.488254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.488226    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv" pod "e2faa273-0952-4c66-a8f5-adb99e72580c" (UID: "e2faa273-0952-4c66-a8f5-adb99e72580c"). InnerVolumeSpecName "kube-api-access-qmcsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:21.491792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.491674    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t" pod "26ca9220-c784-42ec-9d4d-c5dcf240dae1" (UID: "26ca9220-c784-42ec-9d4d-c5dcf240dae1"). InnerVolumeSpecName "kube-api-access-h246t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:21.584565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.584526    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h246t\" (UniqueName: \"kubernetes.io/projected/26ca9220-c784-42ec-9d4d-c5dcf240dae1-kube-api-access-h246t\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:21.584565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.584563    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmcsv\" (UniqueName: \"kubernetes.io/projected/e2faa273-0952-4c66-a8f5-adb99e72580c-kube-api-access-qmcsv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:21.584829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.584579    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shh29\" (UniqueName: \"kubernetes.io/projected/0ccda7bc-8e19-4b5b-a87d-7409c195d182-kube-api-access-shh29\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:21.584829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.584592    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2faa273-0952-4c66-a8f5-adb99e72580c-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:21.584829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.584613    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda7bc-8e19-4b5b-a87d-7409c195d182-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:21.584829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:21.584626    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ca9220-c784-42ec-9d4d-c5dcf240dae1-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:22.112813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.112766    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:22.141479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.138549    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"
Oct 30 12:00:22.141479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.138550    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl" event={"ID":"26ca9220-c784-42ec-9d4d-c5dcf240dae1","Type":"ContainerDied","Data":"0031b587255877703e59981f4f93f253196e2d4f69658f3c9d92452241f72e6d"}
Oct 30 12:00:22.141479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.138697    5060 scope.go:122] "RemoveContainer" containerID="b74ec41e32b824d493d077c13f2e11207556d24eaf0b443b369d382404e2ae56"
Oct 30 12:00:22.159132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.158324    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf" event={"ID":"0ccda7bc-8e19-4b5b-a87d-7409c195d182","Type":"ContainerDied","Data":"a8845eede07591e66fd5cc01abd8f55324aee3ebb1a655e586977dbd8ebac1ff"}
Oct 30 12:00:22.159132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.158476    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"
Oct 30 12:00:22.173133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.173097    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8" event={"ID":"e2faa273-0952-4c66-a8f5-adb99e72580c","Type":"ContainerDied","Data":"ed64f8935e237e02fd7e5ead02a3a022c1e7e75dcc505c77f7f90b6c3aadbdd6"}
Oct 30 12:00:22.173444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.173407    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"
Oct 30 12:00:22.216789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.216758    5060 scope.go:122] "RemoveContainer" containerID="038ffa44ffcf890798b5012744cb95396f8606c1ae236eecfc63a6f28414e5d3"
Oct 30 12:00:22.229064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.229040    5060 scope.go:122] "RemoveContainer" containerID="f9ac2beb787cd210df82e0f2259df1e2582bad6eca22465f5f65f302dd169112"
Oct 30 12:00:22.257013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.256978    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"]
Oct 30 12:00:22.264935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.264903    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-48e1f1-29889360-tb2v8"]
Oct 30 12:00:22.302563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.302528    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"]
Oct 30 12:00:22.309156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.309121    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889360-2pwdl"]
Oct 30 12:00:22.335203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.335162    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" path="/var/lib/kubelet/pods/26ca9220-c784-42ec-9d4d-c5dcf240dae1/volumes"
Oct 30 12:00:22.335721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.335699    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" path="/var/lib/kubelet/pods/e2faa273-0952-4c66-a8f5-adb99e72580c/volumes"
Oct 30 12:00:22.343730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.343700    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"]
Oct 30 12:00:22.352919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:22.352885    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889360-f8svf"]
Oct 30 12:00:23.942441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:23.942390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:23.971508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:23.971472    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:00:24.335732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:24.335702    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" path="/var/lib/kubelet/pods/0ccda7bc-8e19-4b5b-a87d-7409c195d182/volumes"
Oct 30 12:00:25.660867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:25.660835    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:27.364678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:27.364638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:28.209917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.209872    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"]
Oct 30 12:00:28.210423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210408    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210425    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210443    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210461    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210471    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210475    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210485    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.210489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210490    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210498    5060 cpu_manager.go:414] "containerMap: removing container" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210503    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210509    5060 cpu_manager.go:414] "containerMap: removing container" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210514    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210526    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210530    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210535    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210540    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210545    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210551    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210562    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210566    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210573    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210577    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210584    5060 cpu_manager.go:414] "containerMap: removing container" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210588    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210595    5060 cpu_manager.go:414] "containerMap: removing container" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210599    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210606    5060 cpu_manager.go:414] "containerMap: removing container" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210610    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210618    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210622    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210638    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210642    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210648    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.210715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.210653    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213047    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213104    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213117    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213139    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213167    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213176    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213196    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213204    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213213    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213255    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213264    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01420f8-1dde-455c-bad7-ddf9a9e5bbd1" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213279    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213289    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213319    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="55082e7a-26ec-4444-a1e6-83456c739776" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213336    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213350    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213363    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:28.215728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.213370    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:28.224892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.224852    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.235612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.235580    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"]
Oct 30 12:00:28.243052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.243020    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:00:28.243236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.243220    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:00:28.243295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.243279    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:00:28.252325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.252295    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:00:28.370962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.370914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j948m\" (UniqueName: \"kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.371476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.371066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.371476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.371126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.472722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.472631    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j948m\" (UniqueName: \"kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.472881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.472738    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.472881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.472788    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.473388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.473368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.475434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.475412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.496624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.496577    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j948m\" (UniqueName: \"kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m\") pod \"maas-api-revoke-sub-8cb7ec7c-vnbhk\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.546830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.546795    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:28.858483 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:28.858421    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f72832_c1d3_4030_a059_c04ebd1450a2.slice/crio-6776c528867dbfc3fee49f99972ea29d1121eaf668c8c68409c543bfd5fe6255 WatchSource:0}: Error finding container 6776c528867dbfc3fee49f99972ea29d1121eaf668c8c68409c543bfd5fe6255: Status 404 returned error can't find the container with id 6776c528867dbfc3fee49f99972ea29d1121eaf668c8c68409c543bfd5fe6255
Oct 30 12:00:28.859777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:28.859751    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"]
Oct 30 12:00:29.207786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:29.207750    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:29.243277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:29.243240    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk" event={"ID":"63f72832-c1d3-4030-a059-c04ebd1450a2","Type":"ContainerStarted","Data":"a93d6d1573842751bf8b12cf20f04c624022977e152af44ef2eec93e11464615"}
Oct 30 12:00:29.243519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:29.243302    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk" event={"ID":"63f72832-c1d3-4030-a059-c04ebd1450a2","Type":"ContainerStarted","Data":"6776c528867dbfc3fee49f99972ea29d1121eaf668c8c68409c543bfd5fe6255"}
Oct 30 12:00:29.291513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:29.287688    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk" podStartSLOduration=1.287671425 podStartE2EDuration="1.287671425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:29.28426618 +0000 UTC m=+1833.079150675" watchObservedRunningTime="2026-10-30 12:00:29.287671425 +0000 UTC m=+1833.082555918"
Oct 30 12:00:29.367999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:29.367963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:30.254074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:30.254037    5060 generic.go:356] "Generic (PLEG): container finished" podID="63f72832-c1d3-4030-a059-c04ebd1450a2" containerID="a93d6d1573842751bf8b12cf20f04c624022977e152af44ef2eec93e11464615" exitCode=0
Oct 30 12:00:30.254592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:30.254122    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk" event={"ID":"63f72832-c1d3-4030-a059-c04ebd1450a2","Type":"ContainerDied","Data":"a93d6d1573842751bf8b12cf20f04c624022977e152af44ef2eec93e11464615"}
Oct 30 12:00:30.672911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:30.672810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:31.100795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.100758    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:31.436325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.436297    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:31.513758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.513728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca\") pod \"63f72832-c1d3-4030-a059-c04ebd1450a2\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") "
Oct 30 12:00:31.513937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.513832    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca\") pod \"63f72832-c1d3-4030-a059-c04ebd1450a2\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") "
Oct 30 12:00:31.514050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.514024    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m\" (UniqueName: \"kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m\") pod \"63f72832-c1d3-4030-a059-c04ebd1450a2\" (UID: \"63f72832-c1d3-4030-a059-c04ebd1450a2\") "
Oct 30 12:00:31.514174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.514151    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca" pod "63f72832-c1d3-4030-a059-c04ebd1450a2" (UID: "63f72832-c1d3-4030-a059-c04ebd1450a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:31.514633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.514518    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:31.516291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.516268    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca" pod "63f72832-c1d3-4030-a059-c04ebd1450a2" (UID: "63f72832-c1d3-4030-a059-c04ebd1450a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:00:31.516627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.516546    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m" pod "63f72832-c1d3-4030-a059-c04ebd1450a2" (UID: "63f72832-c1d3-4030-a059-c04ebd1450a2"). InnerVolumeSpecName "kube-api-access-j948m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:31.616012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.615963    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f72832-c1d3-4030-a059-c04ebd1450a2-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:31.616012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:31.616010    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j948m\" (UniqueName: \"kubernetes.io/projected/63f72832-c1d3-4030-a059-c04ebd1450a2-kube-api-access-j948m\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:32.151562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.151511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:32.272576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.272546    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"
Oct 30 12:00:32.272576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.272576    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk" event={"ID":"63f72832-c1d3-4030-a059-c04ebd1450a2","Type":"ContainerDied","Data":"6776c528867dbfc3fee49f99972ea29d1121eaf668c8c68409c543bfd5fe6255"}
Oct 30 12:00:32.272815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.272633    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6776c528867dbfc3fee49f99972ea29d1121eaf668c8c68409c543bfd5fe6255"
Oct 30 12:00:32.610621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.610587    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"]
Oct 30 12:00:32.611417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611395    5060 cpu_manager.go:414] "containerMap: removing container" podUID="63f72832-c1d3-4030-a059-c04ebd1450a2" containerName="revoke-keys"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611421    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f72832-c1d3-4030-a059-c04ebd1450a2" containerName="revoke-keys"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611483    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611492    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611520    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611527    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccda7bc-8e19-4b5b-a87d-7409c195d182" containerName="cleanup"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611548    5060 cpu_manager.go:414] "containerMap: removing container" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:32.611559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611556    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ca9220-c784-42ec-9d4d-c5dcf240dae1" containerName="cleanup"
Oct 30 12:00:32.611974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611576    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:32.611974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611583    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:32.611974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611742    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c719e029-9961-4b8f-99cf-53bad8285414" containerName="cleanup"
Oct 30 12:00:32.611974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611767    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2faa273-0952-4c66-a8f5-adb99e72580c" containerName="cleanup"
Oct 30 12:00:32.611974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611779    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f1eff3-c985-438a-9ea1-53abb8ff5833" containerName="cleanup"
Oct 30 12:00:32.611974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.611795    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f72832-c1d3-4030-a059-c04ebd1450a2" containerName="revoke-keys"
Oct 30 12:00:32.616917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.616894    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.646415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.646377    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45f11c-openshift-default-dockercfg-p2dd7\""
Oct 30 12:00:32.650409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.650377    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"]
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.759255    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.763609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.763654    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.763794    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.763843    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rm7\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.763886    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.763959    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.764018    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.764101    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.781513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.759983    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:32.865258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865157    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865227    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865255    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865318    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865357    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rm7\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865499    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865562    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865647    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.865890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865723    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.866066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.865942    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.866350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.866325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.866438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.866380    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.868494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.868446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.870011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.869982    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.891671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.891631    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.891863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.891731    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rm7\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7\") pod \"e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") " pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:32.916026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.915974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:32.931825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:32.931759    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:33.281553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:33.281523    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"]
Oct 30 12:00:33.283642 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:33.283614    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d6c388_395a_494d_bbb2_f5adf053c0de.slice/crio-95d1800501b09c06f33fe4d3675bcd249ce038b591ee79ad6e9b305855d337c1 WatchSource:0}: Error finding container 95d1800501b09c06f33fe4d3675bcd249ce038b591ee79ad6e9b305855d337c1: Status 404 returned error can't find the container with id 95d1800501b09c06f33fe4d3675bcd249ce038b591ee79ad6e9b305855d337c1
Oct 30 12:00:33.286924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:33.286891    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:33.287039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:33.286979    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:33.287113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:33.287039    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:34.293994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:34.293950    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" event={"ID":"21d6c388-395a-494d-bbb2-f5adf053c0de","Type":"ContainerStarted","Data":"2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"}
Oct 30 12:00:34.294552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:34.294034    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" event={"ID":"21d6c388-395a-494d-bbb2-f5adf053c0de","Type":"ContainerStarted","Data":"95d1800501b09c06f33fe4d3675bcd249ce038b591ee79ad6e9b305855d337c1"}
Oct 30 12:00:34.366743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:34.366694    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podStartSLOduration=2.366678775 podStartE2EDuration="2.366678775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:34.351498455 +0000 UTC m=+1838.146382947" watchObservedRunningTime="2026-10-30 12:00:34.366678775 +0000 UTC m=+1838.161563360"
Oct 30 12:00:34.808852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:34.808817    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:34.932870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:34.932832    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:34.939324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:34.939293    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:35.301334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:35.301306    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:35.302512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:35.302487    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:00:36.683597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:36.683515    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:36.849710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:36.849673    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:37.000019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:36.999960    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:37.157711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:37.157675    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:38.603906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.603802    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:38.763646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.763605    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:38.834615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.834574    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"]
Oct 30 12:00:38.860857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.860781    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:38.885250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.885190    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"]
Oct 30 12:00:38.889640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.889602    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f11c\""
Oct 30 12:00:38.928022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.927988    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f11c-dockercfg-bst6h\""
Oct 30 12:00:38.975904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.975866    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:38.976111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:38.976049    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qhh\" (UniqueName: \"kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh\") pod \"payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:39.077087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.077046    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qhh\" (UniqueName: \"kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh\") pod \"payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:39.077271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.077221    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:39.077954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.077933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:39.101359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.101331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qhh\" (UniqueName: \"kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh\") pod \"payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:39.247513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.247483    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:39.398000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.397965    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:39.558187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.558065    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:39.558489 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:39.558445    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73fed7c_73da_4fa7_8cca_4907ea33edc9.slice/crio-edbc244f4fc2d7a70944ec03232a1162eba48634063c94be5a8262062789f1a7 WatchSource:0}: Error finding container edbc244f4fc2d7a70944ec03232a1162eba48634063c94be5a8262062789f1a7: Status 404 returned error can't find the container with id edbc244f4fc2d7a70944ec03232a1162eba48634063c94be5a8262062789f1a7
Oct 30 12:00:39.559808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.559784    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"]
Oct 30 12:00:39.708523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:39.708485    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:40.354086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:40.354049    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng" event={"ID":"e73fed7c-73da-4fa7-8cca-4907ea33edc9","Type":"ContainerStarted","Data":"42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1"}
Oct 30 12:00:40.354304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:40.354113    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng" event={"ID":"e73fed7c-73da-4fa7-8cca-4907ea33edc9","Type":"ContainerStarted","Data":"edbc244f4fc2d7a70944ec03232a1162eba48634063c94be5a8262062789f1a7"}
Oct 30 12:00:40.354304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:40.354253    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:40.396247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:40.396195    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng" podStartSLOduration=2.396179242 podStartE2EDuration="2.396179242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:40.395069751 +0000 UTC m=+1844.189954246" watchObservedRunningTime="2026-10-30 12:00:40.396179242 +0000 UTC m=+1844.191063736"
Oct 30 12:00:41.379775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:41.379742    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:41.946100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:41.946066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:43.088070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:43.088031    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:43.377877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:43.377772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:43.717254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:43.717218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:44.163274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:44.163186    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:45.463080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:45.463048    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:47.355943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.355904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:47.628538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.626269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:47.770717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.770681    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"]
Oct 30 12:00:47.774927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.774910    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.811427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.811398    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:00:47.811803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.811783    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:00:47.811948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.811937    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:00:47.812063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.812053    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:00:47.817521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.817491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:47.854769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.854736    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"]
Oct 30 12:00:47.870464 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.870418    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.870639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.870572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.870639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.870606    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tk6\" (UniqueName: \"kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.971968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.971931    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.972188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.972045    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.972188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.972083    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tk6\" (UniqueName: \"kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.972620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.972585    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:47.974877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:47.974859    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:48.042217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:48.042180    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tk6\" (UniqueName: \"kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6\") pod \"maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:48.158621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:48.158584    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:48.543852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:48.543816    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"]
Oct 30 12:00:48.546642 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:48.546613    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32dd893_4b81_4d64_a59b_55b467e27a47.slice/crio-e1a5d83f0bcc7a26c219c8946fd4121addf4c8a600fe99ce202274d83dcf9338 WatchSource:0}: Error finding container e1a5d83f0bcc7a26c219c8946fd4121addf4c8a600fe99ce202274d83dcf9338: Status 404 returned error can't find the container with id e1a5d83f0bcc7a26c219c8946fd4121addf4c8a600fe99ce202274d83dcf9338
Oct 30 12:00:49.353502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:49.353439    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:49.527014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:49.526977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:49.605549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:49.590545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h" event={"ID":"f32dd893-4b81-4d64-a59b-55b467e27a47","Type":"ContainerStarted","Data":"f57baf9d76f2698dce8df95f47bf790141875eef6d9a42dd153069a316d4e3bc"}
Oct 30 12:00:49.605549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:49.590633    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h" event={"ID":"f32dd893-4b81-4d64-a59b-55b467e27a47","Type":"ContainerStarted","Data":"e1a5d83f0bcc7a26c219c8946fd4121addf4c8a600fe99ce202274d83dcf9338"}
Oct 30 12:00:49.748849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:49.742516    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h" podStartSLOduration=2.742497249 podStartE2EDuration="2.742497249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:49.740255554 +0000 UTC m=+1853.535140064" watchObservedRunningTime="2026-10-30 12:00:49.742497249 +0000 UTC m=+1853.537381744"
Oct 30 12:00:49.859302 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:00:49.859212    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:00:50.384010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:50.383970    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:50.605591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:50.605500    5060 generic.go:356] "Generic (PLEG): container finished" podID="f32dd893-4b81-4d64-a59b-55b467e27a47" containerID="f57baf9d76f2698dce8df95f47bf790141875eef6d9a42dd153069a316d4e3bc" exitCode=0
Oct 30 12:00:50.606178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:50.605848    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h" event={"ID":"f32dd893-4b81-4d64-a59b-55b467e27a47","Type":"ContainerDied","Data":"f57baf9d76f2698dce8df95f47bf790141875eef6d9a42dd153069a316d4e3bc"}
Oct 30 12:00:51.258647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.258603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:51.366144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.365315    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:00:51.822493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.822130    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:51.930666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.930570    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca\") pod \"f32dd893-4b81-4d64-a59b-55b467e27a47\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") "
Oct 30 12:00:51.930849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.930714    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6\" (UniqueName: \"kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6\") pod \"f32dd893-4b81-4d64-a59b-55b467e27a47\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") "
Oct 30 12:00:51.930849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.930751    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca\") pod \"f32dd893-4b81-4d64-a59b-55b467e27a47\" (UID: \"f32dd893-4b81-4d64-a59b-55b467e27a47\") "
Oct 30 12:00:51.930977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.930951    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca" pod "f32dd893-4b81-4d64-a59b-55b467e27a47" (UID: "f32dd893-4b81-4d64-a59b-55b467e27a47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:00:51.931212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.931187    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:51.933321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.933290    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6" pod "f32dd893-4b81-4d64-a59b-55b467e27a47" (UID: "f32dd893-4b81-4d64-a59b-55b467e27a47"). InnerVolumeSpecName "kube-api-access-z6tk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:00:51.933642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:51.933593    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca" pod "f32dd893-4b81-4d64-a59b-55b467e27a47" (UID: "f32dd893-4b81-4d64-a59b-55b467e27a47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:00:52.032915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:52.032878    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6tk6\" (UniqueName: \"kubernetes.io/projected/f32dd893-4b81-4d64-a59b-55b467e27a47-kube-api-access-z6tk6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:52.032915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:52.032911    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32dd893-4b81-4d64-a59b-55b467e27a47-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:00:52.624770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:52.624684    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"
Oct 30 12:00:52.624937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:52.624687    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h" event={"ID":"f32dd893-4b81-4d64-a59b-55b467e27a47","Type":"ContainerDied","Data":"e1a5d83f0bcc7a26c219c8946fd4121addf4c8a600fe99ce202274d83dcf9338"}
Oct 30 12:00:52.624937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:52.624829    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a5d83f0bcc7a26c219c8946fd4121addf4c8a600fe99ce202274d83dcf9338"
Oct 30 12:00:53.078418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:53.078383    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:54.518616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.518582    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 12:00:54.519059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.518635    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 12:00:54.643726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.643692    5060 generic.go:356] "Generic (PLEG): container finished" podID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerID="beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47" exitCode=0
Oct 30 12:00:54.643974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.643768    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" event={"ID":"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea","Type":"ContainerDied","Data":"beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"}
Oct 30 12:00:54.644260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.644238    5060 scope.go:122] "RemoveContainer" containerID="beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"
Oct 30 12:00:54.645859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.645828    5060 generic.go:356] "Generic (PLEG): container finished" podID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerID="dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1" exitCode=0
Oct 30 12:00:54.645966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.645900    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" event={"ID":"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0","Type":"ContainerDied","Data":"dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"}
Oct 30 12:00:54.646022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.645956    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:54.646062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.646027    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:54.646100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.646061    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:54.646263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.646247    5060 scope.go:122] "RemoveContainer" containerID="dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"
Oct 30 12:00:54.647597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.647570    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:54.647715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.647648    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:54.647715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.647697    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:00:54.979980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:54.979943    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:55.134866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:55.134815    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:55.658754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:55.658718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" event={"ID":"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0","Type":"ContainerStarted","Data":"b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e"}
Oct 30 12:00:55.661800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:55.661771    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" event={"ID":"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea","Type":"ContainerStarted","Data":"b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378"}
Oct 30 12:00:56.250580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.250548    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 12:00:56.251046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.251020    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:00:56.251185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.251091    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 12:00:56.501981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.501833    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 12:00:56.502268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.502244    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 12:00:56.502426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.502405    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 12:00:56.920680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:56.920549    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:57.250681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:57.250640    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:00:57.250861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:57.250707    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 12:00:57.501539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:57.499840    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 12:00:57.501539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:57.499897    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 12:00:57.635832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:57.635797    5060 scope.go:122] "RemoveContainer" containerID="d8bf04acd342aea5c3edfc35a8420af0acea3eee47f1edfb47e68b116ab09ae7"
Oct 30 12:00:58.250975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.250797    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:00:58.250975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.250854    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 12:00:58.255921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.255892    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"]
Oct 30 12:00:58.304479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.303513    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"]
Oct 30 12:00:58.519537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.511177    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 12:00:58.519537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.511237    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 12:00:58.553787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.553748    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:58.749095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.749029    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:58.752556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.752529    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"]
Oct 30 12:00:58.753567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.753542    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f32dd893-4b81-4d64-a59b-55b467e27a47" containerName="revoke-keys"
Oct 30 12:00:58.753741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.753728    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32dd893-4b81-4d64-a59b-55b467e27a47" containerName="revoke-keys"
Oct 30 12:00:58.754040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.754027    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32dd893-4b81-4d64-a59b-55b467e27a47" containerName="revoke-keys"
Oct 30 12:00:58.760680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.760659    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"]
Oct 30 12:00:58.760818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.760808    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.776383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.776107    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-e25f4a\"/\"default-dockercfg-kgl7k\""
Oct 30 12:00:58.776383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.776328    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-e25f4a\"/\"praxis-route-e25f4a-kserve-self-signed-certs\""
Oct 30 12:00:58.836506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.812773    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-e25f4a\"/\"kube-root-ca.crt\""
Oct 30 12:00:58.842496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.841512    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.842496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.841590    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.842496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.841645    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.842496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.841681    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdk9\" (UniqueName: \"kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.842496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.841802    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.842496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.841830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.939478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.917217    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-e25f4a\"/\"openshift-service-ca.crt\""
Oct 30 12:00:58.939774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.939743    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:00:58.944271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.944235    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.944487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.944470    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.948891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.948856    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.955485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.949127    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.955792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.955768    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.955957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.955937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdk9\" (UniqueName: \"kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.961490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.950304    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.967484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.966933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.969166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.968501    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.974481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.972696    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:58.974481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:58.973075    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:59.023477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.013645    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdk9\" (UniqueName: \"kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9\") pod \"praxis-route-e25f4a-kserve-54db69b895-5hzkp\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") " pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:59.154535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.153739    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:00:59.256336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.256262    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:00:59.257198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.257074    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 12:00:59.512594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.512552    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 12:00:59.526745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.512927    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 12:00:59.956021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:00:59.937195    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"]
Oct 30 12:00:59.956021 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:00:59.953329    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f9aaed8_04a4_45fe_ac86_656a21e2f422.slice/crio-a8bd3ec19e8cb280d14992f05f5c1e323a4154271105f1a456a270f4541c5118 WatchSource:0}: Error finding container a8bd3ec19e8cb280d14992f05f5c1e323a4154271105f1a456a270f4541c5118: Status 404 returned error can't find the container with id a8bd3ec19e8cb280d14992f05f5c1e323a4154271105f1a456a270f4541c5118
Oct 30 12:01:00.250916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:00.250876    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:01:00.251149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:00.251129    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 12:01:00.505540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:00.502227    5060 patch_prober.go:29] interesting pod/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 12:01:00.505540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:00.502319    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 12:01:00.581125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:00.581087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:01.035232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.035190    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" event={"ID":"5f9aaed8-04a4-45fe-ac86-656a21e2f422","Type":"ContainerStarted","Data":"062532ef244b55cc5e7722f8a8143d119a1736e0845b19771362af3f5d975b1c"}
Oct 30 12:01:01.035422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.035251    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" event={"ID":"5f9aaed8-04a4-45fe-ac86-656a21e2f422","Type":"ContainerStarted","Data":"a8bd3ec19e8cb280d14992f05f5c1e323a4154271105f1a456a270f4541c5118"}
Oct 30 12:01:01.082601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.082548    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" podStartSLOduration=3.082533765 podStartE2EDuration="3.082533765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:01.078410722 +0000 UTC m=+1864.873295217" watchObservedRunningTime="2026-10-30 12:01:01.082533765 +0000 UTC m=+1864.877418259"
Oct 30 12:01:01.154910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.154873    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:01:01.255249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.255220    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 12:01:01.255715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.255684    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 12:01:01.256796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.256773    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 12:01:01.279799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.279758    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"]
Oct 30 12:01:01.284097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.284069    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.299063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.299036    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:01:01.301680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.301653    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"]
Oct 30 12:01:01.302193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.302157    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:01:01.302521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.302499    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:01:01.399335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.399310    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:01:01.469846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.469811    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqv68\" (UniqueName: \"kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.470302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.470277    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.470619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.470599    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.503701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.503667    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 12:01:01.504013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.503993    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 12:01:01.504839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.504821    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 12:01:01.572522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.572400    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqv68\" (UniqueName: \"kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.572522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.572505    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.573027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.572608    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.573269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.573249    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.575099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.575081    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.597356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.597314    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqv68\" (UniqueName: \"kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68\") pod \"maas-api-revoke-sub-076723b0-lvx5w\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.600009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.599980    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:01.923268 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:01:01.923241    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef4f339_d4a4_45cb_9279_fb01c4ae0599.slice/crio-c94caa694d4e4205e2fe0f4b85ea2822419f4a308f0917cc43d639793a251c1e WatchSource:0}: Error finding container c94caa694d4e4205e2fe0f4b85ea2822419f4a308f0917cc43d639793a251c1e: Status 404 returned error can't find the container with id c94caa694d4e4205e2fe0f4b85ea2822419f4a308f0917cc43d639793a251c1e
Oct 30 12:01:01.926873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:01.926848    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"]
Oct 30 12:01:02.045497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:02.045442    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w" event={"ID":"bef4f339-d4a4-45cb-9279-fb01c4ae0599","Type":"ContainerStarted","Data":"c94caa694d4e4205e2fe0f4b85ea2822419f4a308f0917cc43d639793a251c1e"}
Oct 30 12:01:02.895679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:02.895642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:03.055154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.055067    5060 generic.go:356] "Generic (PLEG): container finished" podID="bef4f339-d4a4-45cb-9279-fb01c4ae0599" containerID="afb9e14aa99467fb1be8b39e6e0ac3d370d40ef8f4b9d750bb60ed238e9050d1" exitCode=0
Oct 30 12:01:03.055320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.055162    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w" event={"ID":"bef4f339-d4a4-45cb-9279-fb01c4ae0599","Type":"ContainerDied","Data":"afb9e14aa99467fb1be8b39e6e0ac3d370d40ef8f4b9d750bb60ed238e9050d1"}
Oct 30 12:01:03.256558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.256511    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4" podUID="80dc6036-09f8-4b92-a48d-706884607ba8" containerName="payload-processing" containerID="cri-o://4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6" gracePeriod=25
Oct 30 12:01:03.304194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.304148    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk" podUID="31bb5375-fe10-4dca-8526-d82eef3fb816" containerName="payload-pre-processing" containerID="cri-o://392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c" gracePeriod=25
Oct 30 12:01:03.591875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.591807    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 12:01:03.608152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.608119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume\") pod \"80dc6036-09f8-4b92-a48d-706884607ba8\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") "
Oct 30 12:01:03.608319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.608177    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl\" (UniqueName: \"kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl\") pod \"80dc6036-09f8-4b92-a48d-706884607ba8\" (UID: \"80dc6036-09f8-4b92-a48d-706884607ba8\") "
Oct 30 12:01:03.608536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.608509    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume" pod "80dc6036-09f8-4b92-a48d-706884607ba8" (UID: "80dc6036-09f8-4b92-a48d-706884607ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:03.608894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.608874    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc6036-09f8-4b92-a48d-706884607ba8-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:03.610704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.610682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl" pod "80dc6036-09f8-4b92-a48d-706884607ba8" (UID: "80dc6036-09f8-4b92-a48d-706884607ba8"). InnerVolumeSpecName "kube-api-access-q44dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:03.644500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.644014    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 12:01:03.709632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.709600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume\") pod \"31bb5375-fe10-4dca-8526-d82eef3fb816\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") "
Oct 30 12:01:03.709868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.709744    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr\" (UniqueName: \"kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr\") pod \"31bb5375-fe10-4dca-8526-d82eef3fb816\" (UID: \"31bb5375-fe10-4dca-8526-d82eef3fb816\") "
Oct 30 12:01:03.710078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.710053    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume" pod "31bb5375-fe10-4dca-8526-d82eef3fb816" (UID: "31bb5375-fe10-4dca-8526-d82eef3fb816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:03.710575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.710532    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44dl\" (UniqueName: \"kubernetes.io/projected/80dc6036-09f8-4b92-a48d-706884607ba8-kube-api-access-q44dl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:03.710727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.710578    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bb5375-fe10-4dca-8526-d82eef3fb816-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:03.712179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.712154    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr" pod "31bb5375-fe10-4dca-8526-d82eef3fb816" (UID: "31bb5375-fe10-4dca-8526-d82eef3fb816"). InnerVolumeSpecName "kube-api-access-9m6gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:03.812101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:03.812066    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6gr\" (UniqueName: \"kubernetes.io/projected/31bb5375-fe10-4dca-8526-d82eef3fb816-kube-api-access-9m6gr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:04.064575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.064533    5060 generic.go:356] "Generic (PLEG): container finished" podID="31bb5375-fe10-4dca-8526-d82eef3fb816" containerID="392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c" exitCode=0
Oct 30 12:01:04.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.064585    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"
Oct 30 12:01:04.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.064620    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk" event={"ID":"31bb5375-fe10-4dca-8526-d82eef3fb816","Type":"ContainerDied","Data":"392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c"}
Oct 30 12:01:04.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.064678    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk" event={"ID":"31bb5375-fe10-4dca-8526-d82eef3fb816","Type":"ContainerDied","Data":"2ac31e64a0db840c8bb4dad32d2fe1cf0dcb07152cbe38aa4736bbc781ffe551"}
Oct 30 12:01:04.065085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.064699    5060 scope.go:122] "RemoveContainer" containerID="392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c"
Oct 30 12:01:04.066370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.066346    5060 generic.go:356] "Generic (PLEG): container finished" podID="80dc6036-09f8-4b92-a48d-706884607ba8" containerID="4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6" exitCode=0
Oct 30 12:01:04.066510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.066396    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"
Oct 30 12:01:04.066510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.066426    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4" event={"ID":"80dc6036-09f8-4b92-a48d-706884607ba8","Type":"ContainerDied","Data":"4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6"}
Oct 30 12:01:04.066610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.066509    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4" event={"ID":"80dc6036-09f8-4b92-a48d-706884607ba8","Type":"ContainerDied","Data":"ae9da01000f27ec19c4dfcbd931fddb021b203fe57da511e39546ede85959561"}
Oct 30 12:01:04.084755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.084728    5060 scope.go:122] "RemoveContainer" containerID="392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c"
Oct 30 12:01:04.085227 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:04.085191    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c\": container with ID starting with 392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c not found: ID does not exist" containerID="392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c"
Oct 30 12:01:04.085346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.085238    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c"} err="failed to get container status \"392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c\": rpc error: code = NotFound desc = could not find container \"392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c\": container with ID starting with 392223235cac54e24ac326d789085bc5ef2628e8d16b263225015bd426554d0c not found: ID does not exist"
Oct 30 12:01:04.085346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.085260    5060 scope.go:122] "RemoveContainer" containerID="4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6"
Oct 30 12:01:04.096814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.096785    5060 scope.go:122] "RemoveContainer" containerID="4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6"
Oct 30 12:01:04.097210 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:04.097187    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6\": container with ID starting with 4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6 not found: ID does not exist" containerID="4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6"
Oct 30 12:01:04.097295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.097216    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6"} err="failed to get container status \"4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6\": rpc error: code = NotFound desc = could not find container \"4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6\": container with ID starting with 4d044efbce8561b7e49ce8373757280637e6de476770dc79f82f95c284dcf4c6 not found: ID does not exist"
Oct 30 12:01:04.124725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.124692    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"]
Oct 30 12:01:04.129403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.129364    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b920f1-55cf8579b7-g9jbk"]
Oct 30 12:01:04.168156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.168123    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"]
Oct 30 12:01:04.174248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.174213    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b920f1-55844597bb-bgsl4"]
Oct 30 12:01:04.229550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.229521    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:04.318336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.318238    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca\") pod \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") "
Oct 30 12:01:04.318336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.318281    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca\") pod \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") "
Oct 30 12:01:04.318655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.318563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68\" (UniqueName: \"kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68\") pod \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\" (UID: \"bef4f339-d4a4-45cb-9279-fb01c4ae0599\") "
Oct 30 12:01:04.319710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.319683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca" pod "bef4f339-d4a4-45cb-9279-fb01c4ae0599" (UID: "bef4f339-d4a4-45cb-9279-fb01c4ae0599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:04.321581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.321509    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68" pod "bef4f339-d4a4-45cb-9279-fb01c4ae0599" (UID: "bef4f339-d4a4-45cb-9279-fb01c4ae0599"). InnerVolumeSpecName "kube-api-access-lqv68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:04.321581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.321549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca" pod "bef4f339-d4a4-45cb-9279-fb01c4ae0599" (UID: "bef4f339-d4a4-45cb-9279-fb01c4ae0599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:01:04.336859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.336831    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bb5375-fe10-4dca-8526-d82eef3fb816" path="/var/lib/kubelet/pods/31bb5375-fe10-4dca-8526-d82eef3fb816/volumes"
Oct 30 12:01:04.337246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.337234    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dc6036-09f8-4b92-a48d-706884607ba8" path="/var/lib/kubelet/pods/80dc6036-09f8-4b92-a48d-706884607ba8/volumes"
Oct 30 12:01:04.420231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.420194    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqv68\" (UniqueName: \"kubernetes.io/projected/bef4f339-d4a4-45cb-9279-fb01c4ae0599-kube-api-access-lqv68\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:04.420231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.420225    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:04.420563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.420243    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef4f339-d4a4-45cb-9279-fb01c4ae0599-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:04.654850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:04.654814    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:05.076945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.076914    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"
Oct 30 12:01:05.077470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.076954    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w" event={"ID":"bef4f339-d4a4-45cb-9279-fb01c4ae0599","Type":"ContainerDied","Data":"c94caa694d4e4205e2fe0f4b85ea2822419f4a308f0917cc43d639793a251c1e"}
Oct 30 12:01:05.077470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.077001    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94caa694d4e4205e2fe0f4b85ea2822419f4a308f0917cc43d639793a251c1e"
Oct 30 12:01:05.477419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.477380    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"]
Oct 30 12:01:05.478240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478209    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bef4f339-d4a4-45cb-9279-fb01c4ae0599" containerName="revoke-keys"
Oct 30 12:01:05.478240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478235    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef4f339-d4a4-45cb-9279-fb01c4ae0599" containerName="revoke-keys"
Oct 30 12:01:05.478443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478271    5060 cpu_manager.go:414] "containerMap: removing container" podUID="80dc6036-09f8-4b92-a48d-706884607ba8" containerName="payload-processing"
Oct 30 12:01:05.478443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478279    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc6036-09f8-4b92-a48d-706884607ba8" containerName="payload-processing"
Oct 30 12:01:05.478443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478296    5060 cpu_manager.go:414] "containerMap: removing container" podUID="31bb5375-fe10-4dca-8526-d82eef3fb816" containerName="payload-pre-processing"
Oct 30 12:01:05.478443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478304    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bb5375-fe10-4dca-8526-d82eef3fb816" containerName="payload-pre-processing"
Oct 30 12:01:05.478443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478469    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dc6036-09f8-4b92-a48d-706884607ba8" containerName="payload-processing"
Oct 30 12:01:05.478443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478482    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bb5375-fe10-4dca-8526-d82eef3fb816" containerName="payload-pre-processing"
Oct 30 12:01:05.478853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.478502    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef4f339-d4a4-45cb-9279-fb01c4ae0599" containerName="revoke-keys"
Oct 30 12:01:05.488501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.488469    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.499640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.499571    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"]
Oct 30 12:01:05.510004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.509975    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1d9b1a-openshift-default-dockercfg-p2zdz\""
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.533901    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.533968    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534003    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534038    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534097    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkv79\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.534845    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"]
Oct 30 12:01:05.535166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.535001    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.536153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.536123    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"]
Oct 30 12:01:05.636060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636024    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636068    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636096    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdp8\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636242    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636297    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636349    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636397    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636421    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636507    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636551    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636618    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636722    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636754    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636836    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkv79\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636886    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.636903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636895    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.637184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.636939    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.637184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.637162    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.637184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.637179    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.639075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.639058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.639394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.639366    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.663628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.663595    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.663767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.663697    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkv79\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.738124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738039    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738114    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738137    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738246    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738265    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdp8\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738315    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738357    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738639    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.738777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.738650    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.739045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.739021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.739146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.739100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.739344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.739322    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.740622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.740591    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.740771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.740756    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.762739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.762708    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.762877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.762860    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdp8\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:05.835128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.835071    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:05.853725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:05.853694    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:06.167146 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:01:06.167116    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01871d97_ee7d_44da_b5f6_a462c8826a94.slice/crio-5a5626a382d12a70e0081b593a65777f46ab4518c44a1fd195998b2e9e5d59f8 WatchSource:0}: Error finding container 5a5626a382d12a70e0081b593a65777f46ab4518c44a1fd195998b2e9e5d59f8: Status 404 returned error can't find the container with id 5a5626a382d12a70e0081b593a65777f46ab4518c44a1fd195998b2e9e5d59f8
Oct 30 12:01:06.169033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.169010    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"]
Oct 30 12:01:06.169789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.169759    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:06.169849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.169821    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:06.169887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.169852    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:06.211666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.211637    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"]
Oct 30 12:01:06.213256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.213215    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:06.213376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.213305    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:06.213376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.213345    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:06.375778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:06.375734    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:07.146494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.146422    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" event={"ID":"01871d97-ee7d-44da-b5f6-a462c8826a94","Type":"ContainerStarted","Data":"a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f"}
Oct 30 12:01:07.147611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.147550    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" event={"ID":"01871d97-ee7d-44da-b5f6-a462c8826a94","Type":"ContainerStarted","Data":"5a5626a382d12a70e0081b593a65777f46ab4518c44a1fd195998b2e9e5d59f8"}
Oct 30 12:01:07.149217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.149188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" event={"ID":"eebef590-a0f1-49eb-b964-b4dda45ef640","Type":"ContainerStarted","Data":"3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021"}
Oct 30 12:01:07.149343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.149243    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" event={"ID":"eebef590-a0f1-49eb-b964-b4dda45ef640","Type":"ContainerStarted","Data":"cba54aaba9c4842e74a8cd7954df6b02136a4b3b2a3680f2e6a7b29781a2adb5"}
Oct 30 12:01:07.245133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.245079    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podStartSLOduration=2.24506214 podStartE2EDuration="2.24506214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:07.229136922 +0000 UTC m=+1871.024021432" watchObservedRunningTime="2026-10-30 12:01:07.24506214 +0000 UTC m=+1871.039946634"
Oct 30 12:01:07.310714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.294355    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podStartSLOduration=2.294335314 podStartE2EDuration="2.294335314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:07.27510349 +0000 UTC m=+1871.069987983" watchObservedRunningTime="2026-10-30 12:01:07.294335314 +0000 UTC m=+1871.089219809"
Oct 30 12:01:07.628481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.624520    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:07.772925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.772886    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:07.841588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.839732    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:07.841588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.839862    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:07.841588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.839920    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:07.853953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.853927    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:07.855347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.855305    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:07.855489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:07.855363    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:08.246352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:08.246210    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:08.838504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:08.837710    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:08.838504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:08.837785    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:08.854131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:08.854097    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:08.854306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:08.854156    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:09.155850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.155403    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:01:09.168474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.168424    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:01:09.246472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.246159    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:01:09.838915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.838879    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:09.839227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.839200    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:09.854606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.854576    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:09.854748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:09.854628    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:10.044500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:10.044445    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:10.219479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:10.204335    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:10.836477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:10.835880    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:10.836477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:10.835947    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:10.857480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:10.854358    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:10.857480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:10.854418    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:11.659519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:11.646101    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:11.904480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:11.892736    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:11.904480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:11.892798    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:11.904480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:11.894094    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:11.904480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:11.894137    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:12.241139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.241103    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:12.368325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.368288    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"]
Oct 30 12:01:12.374300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.374269    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b920f1-gtx5h"]
Oct 30 12:01:12.405514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.404899    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:12.591413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.591307    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:12.836394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.836357    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:12.836615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.836408    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:12.854132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.854047    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:12.854132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.854097    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:12.909943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:12.909907    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:13.839905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:13.839863    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 12:01:13.840189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:13.840159    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 12:01:13.856963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:13.856923    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 12:01:13.857251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:13.857221    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 12:01:14.355480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:14.355385    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32dd893-4b81-4d64-a59b-55b467e27a47" path="/var/lib/kubelet/pods/f32dd893-4b81-4d64-a59b-55b467e27a47/volumes"
Oct 30 12:01:14.920324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:14.920161    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:15.263566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:15.260657    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:15.263566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:15.261615    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:15.263566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:15.262840    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:01:15.474706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:15.474673    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:15.480741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:15.480719    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:15.502787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:15.502756    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:01:16.755417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:16.755311    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:18.461468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:18.461425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:19.214484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:19.214417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:19.354915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:19.354881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:19.860153 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:19.860116    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:01:20.266594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:20.266550    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:21.344611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:21.344562    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:01:21.345243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:21.344626    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:01:22.049117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:22.049080    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:24.093206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.093168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:24.471242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.471210    5060 generic.go:356] "Generic (PLEG): container finished" podID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerID="2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72" exitCode=0
Oct 30 12:01:24.471474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.471281    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" event={"ID":"21d6c388-395a-494d-bbb2-f5adf053c0de","Type":"ContainerDied","Data":"2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"}
Oct 30 12:01:24.471770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.471754    5060 scope.go:122] "RemoveContainer" containerID="2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"
Oct 30 12:01:24.473355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.473327    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:24.473496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.473399    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:24.473496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:24.473431    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:01:25.490612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:25.490575    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" event={"ID":"21d6c388-395a-494d-bbb2-f5adf053c0de","Type":"ContainerStarted","Data":"0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b"}
Oct 30 12:01:25.932339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:25.932243    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:01:25.932580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:25.932535    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 12:01:25.932664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:25.932588    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 12:01:26.559354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:26.559317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:26.932240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:26.932144    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 12:01:26.932240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:26.932205    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 12:01:27.212431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:27.212396    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"]
Oct 30 12:01:27.411570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:27.411488    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:27.712359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:27.712321    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:27.901581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:27.897114    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:27.932209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:27.932171    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 12:01:27.932424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:27.932248    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 12:01:28.489476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:28.485925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:28.933000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:28.932906    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 12:01:28.933000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:28.932979    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 12:01:29.503482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:29.503378    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"]
Oct 30 12:01:29.503709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:29.503658    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy" containerID="cri-o://77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902" gracePeriod=30
Oct 30 12:01:29.557252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:29.557214    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:29.933157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:29.933062    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 12:01:29.933157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:29.933114    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 12:01:30.254917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:30.254882    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:30.936810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:30.936782    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:01:30.937320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:30.937279    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:01:30.938061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:30.938043    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:01:31.955337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.955305    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"]
Oct 30 12:01:31.960464 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.960421    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:31.960783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.960769    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:31.993728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.993681    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"]
Oct 30 12:01:31.994000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.993975    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:01:31.994120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.994028    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:01:31.994120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.994099    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:01:31.994238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:31.994155    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:01:32.032116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.032082    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.032360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.032237    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.032360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.032328    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx949\" (UniqueName: \"kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.133545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.133502    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.133762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.133567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx949\" (UniqueName: \"kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.133762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.133669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.134313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.134287    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.136486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.136424    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.158002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.157971    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx949\" (UniqueName: \"kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949\") pod \"maas-api-revoke-sub-d980dcdd-rbjm7\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.213171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.213061    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng" podUID="e73fed7c-73da-4fa7-8cca-4907ea33edc9" containerName="payload-processing" containerID="cri-o://42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1" gracePeriod=25
Oct 30 12:01:32.296694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.296660    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:32.528033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.528003    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:01:32.581846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.581812    5060 generic.go:356] "Generic (PLEG): container finished" podID="e73fed7c-73da-4fa7-8cca-4907ea33edc9" containerID="42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1" exitCode=0
Oct 30 12:01:32.582036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.581870    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"
Oct 30 12:01:32.582036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.581887    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng" event={"ID":"e73fed7c-73da-4fa7-8cca-4907ea33edc9","Type":"ContainerDied","Data":"42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1"}
Oct 30 12:01:32.582036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.581949    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng" event={"ID":"e73fed7c-73da-4fa7-8cca-4907ea33edc9","Type":"ContainerDied","Data":"edbc244f4fc2d7a70944ec03232a1162eba48634063c94be5a8262062789f1a7"}
Oct 30 12:01:32.582036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.581978    5060 scope.go:122] "RemoveContainer" containerID="42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1"
Oct 30 12:01:32.595243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.595213    5060 scope.go:122] "RemoveContainer" containerID="42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1"
Oct 30 12:01:32.595657 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:32.595634    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1\": container with ID starting with 42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1 not found: ID does not exist" containerID="42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1"
Oct 30 12:01:32.595763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.595668    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1"} err="failed to get container status \"42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1\": rpc error: code = NotFound desc = could not find container \"42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1\": container with ID starting with 42f0cf502ba8d592fecfcf59ebc4f64686ee9a792298688d6f3ffd12023e48e1 not found: ID does not exist"
Oct 30 12:01:32.608653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.608627    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"]
Oct 30 12:01:32.639832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.639806    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume\") pod \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") "
Oct 30 12:01:32.639952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.639919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh\" (UniqueName: \"kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh\") pod \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\" (UID: \"e73fed7c-73da-4fa7-8cca-4907ea33edc9\") "
Oct 30 12:01:32.640218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.640190    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume" pod "e73fed7c-73da-4fa7-8cca-4907ea33edc9" (UID: "e73fed7c-73da-4fa7-8cca-4907ea33edc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:32.640392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.640364    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73fed7c-73da-4fa7-8cca-4907ea33edc9-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:32.642154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.642133    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh" pod "e73fed7c-73da-4fa7-8cca-4907ea33edc9" (UID: "e73fed7c-73da-4fa7-8cca-4907ea33edc9"). InnerVolumeSpecName "kube-api-access-d8qhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:32.741993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.741950    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qhh\" (UniqueName: \"kubernetes.io/projected/e73fed7c-73da-4fa7-8cca-4907ea33edc9-kube-api-access-d8qhh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:32.933979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.933946    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"]
Oct 30 12:01:32.939621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:32.939593    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-88685759d-hlhng"]
Oct 30 12:01:33.594052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:33.594020    5060 generic.go:356] "Generic (PLEG): container finished" podID="ed6c2540-ad15-4451-80a7-a6ef26108a77" containerID="c79b5947070a2797d8d182b1d8e5c6170f67482b379b15a6dce6cc9e45d9f85f" exitCode=0
Oct 30 12:01:33.594428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:33.594096    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7" event={"ID":"ed6c2540-ad15-4451-80a7-a6ef26108a77","Type":"ContainerDied","Data":"c79b5947070a2797d8d182b1d8e5c6170f67482b379b15a6dce6cc9e45d9f85f"}
Oct 30 12:01:33.594428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:33.594158    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7" event={"ID":"ed6c2540-ad15-4451-80a7-a6ef26108a77","Type":"ContainerStarted","Data":"0fa83240fb8793e0b9ebf547783cf604e58d46486b700f1c91a9ba325273660d"}
Oct 30 12:01:33.776513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:33.776477    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:34.335714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.335678    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73fed7c-73da-4fa7-8cca-4907ea33edc9" path="/var/lib/kubelet/pods/e73fed7c-73da-4fa7-8cca-4907ea33edc9/volumes"
Oct 30 12:01:34.573382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.573352    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"]
Oct 30 12:01:34.573959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.573945    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e73fed7c-73da-4fa7-8cca-4907ea33edc9" containerName="payload-processing"
Oct 30 12:01:34.574020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.573961    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73fed7c-73da-4fa7-8cca-4907ea33edc9" containerName="payload-processing"
Oct 30 12:01:34.574092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.574083    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73fed7c-73da-4fa7-8cca-4907ea33edc9" containerName="payload-processing"
Oct 30 12:01:34.577509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.577487    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.598824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.597555    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"]
Oct 30 12:01:34.666306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.666274    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jbl\" (UniqueName: \"kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.666550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.666524    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.666699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.666676    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.769877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.768780    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.769877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.768936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.769877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.768976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jbl\" (UniqueName: \"kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.771099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.771072    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.778356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.778319    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.822164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.822111    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jbl\" (UniqueName: \"kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl\") pod \"maas-api-revoke-sub-9edb0a08-pjb5f\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.841975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.841943    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:34.860925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.860897    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 12:01:34.943109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.943073    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:34.971427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971394    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971446    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971512    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971551    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971603    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca\") pod \"ed6c2540-ad15-4451-80a7-a6ef26108a77\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") "
Oct 30 12:01:34.971936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971646    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971682    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971806    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971853    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.971936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971894    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca\") pod \"ed6c2540-ad15-4451-80a7-a6ef26108a77\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") "
Oct 30 12:01:34.972205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971975    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert\") pod \"0952119e-66dd-4e14-95f0-f5971be5c762\" (UID: \"0952119e-66dd-4e14-95f0-f5971be5c762\") "
Oct 30 12:01:34.972205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.971976    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:01:34.972205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972020    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949\" (UniqueName: \"kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949\") pod \"ed6c2540-ad15-4451-80a7-a6ef26108a77\" (UID: \"ed6c2540-ad15-4451-80a7-a6ef26108a77\") "
Oct 30 12:01:34.972205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972177    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:01:34.972426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972381    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:01:34.972426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972399    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca" pod "ed6c2540-ad15-4451-80a7-a6ef26108a77" (UID: "ed6c2540-ad15-4451-80a7-a6ef26108a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:34.972643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:01:34.972763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972749    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:34.972827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972769    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:34.972827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972784    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:34.972939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972828    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:34.972939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972842    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:34.973054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.972980    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:34.974755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.974725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:34.975253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.975227    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949" pod "ed6c2540-ad15-4451-80a7-a6ef26108a77" (UID: "ed6c2540-ad15-4451-80a7-a6ef26108a77"). InnerVolumeSpecName "kube-api-access-dx949". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:34.975925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.975902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:01:34.976098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.976080    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:01:34.976192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.976173    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4" pod "0952119e-66dd-4e14-95f0-f5971be5c762" (UID: "0952119e-66dd-4e14-95f0-f5971be5c762"). InnerVolumeSpecName "kube-api-access-fkzl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:34.977223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:34.977203    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca" pod "ed6c2540-ad15-4451-80a7-a6ef26108a77" (UID: "ed6c2540-ad15-4451-80a7-a6ef26108a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:01:35.074092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074055    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0952119e-66dd-4e14-95f0-f5971be5c762-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.074092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074087    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6c2540-ad15-4451-80a7-a6ef26108a77-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.074092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074096    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0952119e-66dd-4e14-95f0-f5971be5c762-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.074427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074104    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx949\" (UniqueName: \"kubernetes.io/projected/ed6c2540-ad15-4451-80a7-a6ef26108a77-kube-api-access-dx949\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.074427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074115    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkzl4\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-kube-api-access-fkzl4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.074427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074122    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0952119e-66dd-4e14-95f0-f5971be5c762-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.074427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.074131    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0952119e-66dd-4e14-95f0-f5971be5c762-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:35.295014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.294956    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"]
Oct 30 12:01:35.627294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.627167    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:35.635627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.635593    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"
Oct 30 12:01:35.635835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.635777    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7" event={"ID":"ed6c2540-ad15-4451-80a7-a6ef26108a77","Type":"ContainerDied","Data":"0fa83240fb8793e0b9ebf547783cf604e58d46486b700f1c91a9ba325273660d"}
Oct 30 12:01:35.635917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.635834    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa83240fb8793e0b9ebf547783cf604e58d46486b700f1c91a9ba325273660d"
Oct 30 12:01:35.638637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.638606    5060 generic.go:356] "Generic (PLEG): container finished" podID="0952119e-66dd-4e14-95f0-f5971be5c762" containerID="77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902" exitCode=0
Oct 30 12:01:35.638789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.638705    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"
Oct 30 12:01:35.638789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.638756    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" event={"ID":"0952119e-66dd-4e14-95f0-f5971be5c762","Type":"ContainerDied","Data":"77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902"}
Oct 30 12:01:35.638918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.638807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p" event={"ID":"0952119e-66dd-4e14-95f0-f5971be5c762","Type":"ContainerDied","Data":"cf39234789136cc67b71b4865d90e5bd67829041271933806eb9e5805c05f576"}
Oct 30 12:01:35.638918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.638838    5060 scope.go:122] "RemoveContainer" containerID="77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902"
Oct 30 12:01:35.642616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.641864    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f" event={"ID":"acb955f2-055e-4bd3-bf2f-de8027cf7892","Type":"ContainerStarted","Data":"0c1dcdb05d11ee2805b8b652daf4b0178dd9ba70a954577358913caa410b1a34"}
Oct 30 12:01:35.642616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.641916    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f" event={"ID":"acb955f2-055e-4bd3-bf2f-de8027cf7892","Type":"ContainerStarted","Data":"71607e8378833227bd6c8e8a73a3fde20dc0d593d8471e0e5903b6d170566646"}
Oct 30 12:01:35.655255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.655236    5060 scope.go:122] "RemoveContainer" containerID="71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"
Oct 30 12:01:35.666551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.666524    5060 scope.go:122] "RemoveContainer" containerID="77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902"
Oct 30 12:01:35.666924 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:35.666899    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902\": container with ID starting with 77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902 not found: ID does not exist" containerID="77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902"
Oct 30 12:01:35.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.666935    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902"} err="failed to get container status \"77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902\": rpc error: code = NotFound desc = could not find container \"77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902\": container with ID starting with 77d998775c8823a660bd978ceb4fbaa369a72c5480409b88cbc4215fa0449902 not found: ID does not exist"
Oct 30 12:01:35.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.666954    5060 scope.go:122] "RemoveContainer" containerID="71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"
Oct 30 12:01:35.667226 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:35.667207    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70\": container with ID starting with 71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70 not found: ID does not exist" containerID="71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"
Oct 30 12:01:35.667294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.667235    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70"} err="failed to get container status \"71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70\": rpc error: code = NotFound desc = could not find container \"71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70\": container with ID starting with 71de4aaa4567a153b3037c29b0173e9013087ecc5da06fe306abbde054120e70 not found: ID does not exist"
Oct 30 12:01:35.695955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.695906    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f" podStartSLOduration=1.6958926970000001 podStartE2EDuration="1.695892697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:35.695645282 +0000 UTC m=+1899.490529776" watchObservedRunningTime="2026-10-30 12:01:35.695892697 +0000 UTC m=+1899.490777191"
Oct 30 12:01:35.739036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.739004    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"]
Oct 30 12:01:35.748934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:35.748904    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b920f1-openshift-default-6f7c86785c-lh47p"]
Oct 30 12:01:36.335714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:36.335680    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" path="/var/lib/kubelet/pods/0952119e-66dd-4e14-95f0-f5971be5c762/volumes"
Oct 30 12:01:36.652695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:36.652606    5060 generic.go:356] "Generic (PLEG): container finished" podID="acb955f2-055e-4bd3-bf2f-de8027cf7892" containerID="0c1dcdb05d11ee2805b8b652daf4b0178dd9ba70a954577358913caa410b1a34" exitCode=0
Oct 30 12:01:36.652695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:36.652683    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f" event={"ID":"acb955f2-055e-4bd3-bf2f-de8027cf7892","Type":"ContainerDied","Data":"0c1dcdb05d11ee2805b8b652daf4b0178dd9ba70a954577358913caa410b1a34"}
Oct 30 12:01:37.535359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.535319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:37.827105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.827079    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:37.913425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.913388    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca\") pod \"acb955f2-055e-4bd3-bf2f-de8027cf7892\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") "
Oct 30 12:01:37.913645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.913511    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca\") pod \"acb955f2-055e-4bd3-bf2f-de8027cf7892\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") "
Oct 30 12:01:37.913749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.913708    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl\" (UniqueName: \"kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl\") pod \"acb955f2-055e-4bd3-bf2f-de8027cf7892\" (UID: \"acb955f2-055e-4bd3-bf2f-de8027cf7892\") "
Oct 30 12:01:37.914347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.914312    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca" pod "acb955f2-055e-4bd3-bf2f-de8027cf7892" (UID: "acb955f2-055e-4bd3-bf2f-de8027cf7892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:37.916483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.916424    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl" pod "acb955f2-055e-4bd3-bf2f-de8027cf7892" (UID: "acb955f2-055e-4bd3-bf2f-de8027cf7892"). InnerVolumeSpecName "kube-api-access-s4jbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:37.916483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:37.916426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca" pod "acb955f2-055e-4bd3-bf2f-de8027cf7892" (UID: "acb955f2-055e-4bd3-bf2f-de8027cf7892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:01:38.015377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.015336    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4jbl\" (UniqueName: \"kubernetes.io/projected/acb955f2-055e-4bd3-bf2f-de8027cf7892-kube-api-access-s4jbl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:38.015377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.015370    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:38.015377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.015380    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acb955f2-055e-4bd3-bf2f-de8027cf7892-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:38.535889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.535855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:38.669392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.669317    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"
Oct 30 12:01:38.669392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.669352    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f" event={"ID":"acb955f2-055e-4bd3-bf2f-de8027cf7892","Type":"ContainerDied","Data":"71607e8378833227bd6c8e8a73a3fde20dc0d593d8471e0e5903b6d170566646"}
Oct 30 12:01:38.669601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:38.669401    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71607e8378833227bd6c8e8a73a3fde20dc0d593d8471e0e5903b6d170566646"
Oct 30 12:01:39.442562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:39.442527    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:41.274272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:41.274238    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:43.544925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:43.544885    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:44.466716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.466681    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"]
Oct 30 12:01:44.467225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467213    5060 cpu_manager.go:414] "containerMap: removing container" podUID="acb955f2-055e-4bd3-bf2f-de8027cf7892" containerName="revoke-keys"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467227    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb955f2-055e-4bd3-bf2f-de8027cf7892" containerName="revoke-keys"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467244    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467250    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467263    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467269    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467279    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ed6c2540-ad15-4451-80a7-a6ef26108a77" containerName="revoke-keys"
Oct 30 12:01:44.467285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467283    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6c2540-ad15-4451-80a7-a6ef26108a77" containerName="revoke-keys"
Oct 30 12:01:44.467540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467383    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb955f2-055e-4bd3-bf2f-de8027cf7892" containerName="revoke-keys"
Oct 30 12:01:44.467540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467392    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy"
Oct 30 12:01:44.467540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467400    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0952119e-66dd-4e14-95f0-f5971be5c762" containerName="istio-proxy"
Oct 30 12:01:44.467540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.467406    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6c2540-ad15-4451-80a7-a6ef26108a77" containerName="revoke-keys"
Oct 30 12:01:44.471141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.471123    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.488302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.488269    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"]
Oct 30 12:01:44.496238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.496210    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:01:44.496925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.496906    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:01:44.497056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.497041    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:01:44.498510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.498492    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:01:44.625891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.625844    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.626314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.626032    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnb6\" (UniqueName: \"kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.626314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.626087    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.728041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.727938    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.728206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.728053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.728206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.728159    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnb6\" (UniqueName: \"kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.728916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.728833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.730573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.730553    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.751996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.751968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnb6\" (UniqueName: \"kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6\") pod \"maas-api-revoke-sub-ebd2e292-cjdxw\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.786210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.786165    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:44.935707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.935662    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"]
Oct 30 12:01:44.940792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.940773    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:44.956701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:44.956679    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"]
Oct 30 12:01:45.098676 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:01:45.098642    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd190f090_23ff_46be_85f0_c001db92d9e7.slice/crio-1dd5bd22a977b8c2db850b1a3771c86ebf9b6923943a31a02e3b30e03e470e41 WatchSource:0}: Error finding container 1dd5bd22a977b8c2db850b1a3771c86ebf9b6923943a31a02e3b30e03e470e41: Status 404 returned error can't find the container with id 1dd5bd22a977b8c2db850b1a3771c86ebf9b6923943a31a02e3b30e03e470e41
Oct 30 12:01:45.099332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.099307    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"]
Oct 30 12:01:45.132659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.132628    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.132812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.132786    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.133038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.132827    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdcvb\" (UniqueName: \"kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.233847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.233808    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.234036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.233902    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.234036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.233936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdcvb\" (UniqueName: \"kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.234553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.234519    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.236773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.236749    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.257155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.257131    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdcvb\" (UniqueName: \"kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb\") pod \"maas-api-revoke-sub-5ceb824e-jnhzm\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.533925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.533895    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:45.557825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.556783    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:45.762444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.762392    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw" event={"ID":"d190f090-23ff-46be-85f0-c001db92d9e7","Type":"ContainerStarted","Data":"73cd1240d98c9acabec8e71847748fa6f251144d6cd08583ebfabca0c2a87e13"}
Oct 30 12:01:45.762951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.762631    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw" event={"ID":"d190f090-23ff-46be-85f0-c001db92d9e7","Type":"ContainerStarted","Data":"1dd5bd22a977b8c2db850b1a3771c86ebf9b6923943a31a02e3b30e03e470e41"}
Oct 30 12:01:45.808663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.808589    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw" podStartSLOduration=1.8085722180000001 podStartE2EDuration="1.808572218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:45.803444455 +0000 UTC m=+1909.598328949" watchObservedRunningTime="2026-10-30 12:01:45.808572218 +0000 UTC m=+1909.603456713"
Oct 30 12:01:45.870540 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:01:45.870507    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f86870_543e_4547_a11b_8c2d401f75b6.slice/crio-2c097edd36343ab9720ebe676e9c761b9d295ff932f3ce7097507bf52e10a80d WatchSource:0}: Error finding container 2c097edd36343ab9720ebe676e9c761b9d295ff932f3ce7097507bf52e10a80d: Status 404 returned error can't find the container with id 2c097edd36343ab9720ebe676e9c761b9d295ff932f3ce7097507bf52e10a80d
Oct 30 12:01:45.871510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:45.871481    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"]
Oct 30 12:01:46.774423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:46.774382    5060 generic.go:356] "Generic (PLEG): container finished" podID="d190f090-23ff-46be-85f0-c001db92d9e7" containerID="73cd1240d98c9acabec8e71847748fa6f251144d6cd08583ebfabca0c2a87e13" exitCode=0
Oct 30 12:01:46.774917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:46.774498    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw" event={"ID":"d190f090-23ff-46be-85f0-c001db92d9e7","Type":"ContainerDied","Data":"73cd1240d98c9acabec8e71847748fa6f251144d6cd08583ebfabca0c2a87e13"}
Oct 30 12:01:46.777168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:46.777120    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm" event={"ID":"35f86870-543e-4547-a11b-8c2d401f75b6","Type":"ContainerStarted","Data":"7fdae5b50e85c4091fb4e70f41196ac73afa767def35fc9f825e37ae674d1579"}
Oct 30 12:01:46.777321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:46.777201    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm" event={"ID":"35f86870-543e-4547-a11b-8c2d401f75b6","Type":"ContainerStarted","Data":"2c097edd36343ab9720ebe676e9c761b9d295ff932f3ce7097507bf52e10a80d"}
Oct 30 12:01:46.858825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:46.858754    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm" podStartSLOduration=2.8587342529999997 podStartE2EDuration="2.858734253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:46.857963327 +0000 UTC m=+1910.652847820" watchObservedRunningTime="2026-10-30 12:01:46.858734253 +0000 UTC m=+1910.653618748"
Oct 30 12:01:46.859240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:46.859218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:47.036304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.036208    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:47.788958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.788915    5060 generic.go:356] "Generic (PLEG): container finished" podID="35f86870-543e-4547-a11b-8c2d401f75b6" containerID="7fdae5b50e85c4091fb4e70f41196ac73afa767def35fc9f825e37ae674d1579" exitCode=0
Oct 30 12:01:47.789506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.789005    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm" event={"ID":"35f86870-543e-4547-a11b-8c2d401f75b6","Type":"ContainerDied","Data":"7fdae5b50e85c4091fb4e70f41196ac73afa767def35fc9f825e37ae674d1579"}
Oct 30 12:01:47.953115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.953080    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:47.982726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.982686    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca\") pod \"d190f090-23ff-46be-85f0-c001db92d9e7\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") "
Oct 30 12:01:47.982915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.982789    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca\") pod \"d190f090-23ff-46be-85f0-c001db92d9e7\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") "
Oct 30 12:01:47.982915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.982900    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6\" (UniqueName: \"kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6\") pod \"d190f090-23ff-46be-85f0-c001db92d9e7\" (UID: \"d190f090-23ff-46be-85f0-c001db92d9e7\") "
Oct 30 12:01:47.983147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.983121    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca" pod "d190f090-23ff-46be-85f0-c001db92d9e7" (UID: "d190f090-23ff-46be-85f0-c001db92d9e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:47.983745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.983716    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:47.985206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.985178    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca" pod "d190f090-23ff-46be-85f0-c001db92d9e7" (UID: "d190f090-23ff-46be-85f0-c001db92d9e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:01:47.985387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:47.985364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6" pod "d190f090-23ff-46be-85f0-c001db92d9e7" (UID: "d190f090-23ff-46be-85f0-c001db92d9e7"). InnerVolumeSpecName "kube-api-access-qnnb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:48.085446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.085353    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d190f090-23ff-46be-85f0-c001db92d9e7-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:48.085446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.085388    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnnb6\" (UniqueName: \"kubernetes.io/projected/d190f090-23ff-46be-85f0-c001db92d9e7-kube-api-access-qnnb6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:48.280618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.280545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:48.800096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.799816    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"
Oct 30 12:01:48.800096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.799843    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw" event={"ID":"d190f090-23ff-46be-85f0-c001db92d9e7","Type":"ContainerDied","Data":"1dd5bd22a977b8c2db850b1a3771c86ebf9b6923943a31a02e3b30e03e470e41"}
Oct 30 12:01:48.800096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.799905    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dd5bd22a977b8c2db850b1a3771c86ebf9b6923943a31a02e3b30e03e470e41"
Oct 30 12:01:48.981196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.981171    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:48.999740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.999710    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca\") pod \"35f86870-543e-4547-a11b-8c2d401f75b6\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") "
Oct 30 12:01:48.999937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.999772    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb\" (UniqueName: \"kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb\") pod \"35f86870-543e-4547-a11b-8c2d401f75b6\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") "
Oct 30 12:01:48.999937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:48.999894    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca\") pod \"35f86870-543e-4547-a11b-8c2d401f75b6\" (UID: \"35f86870-543e-4547-a11b-8c2d401f75b6\") "
Oct 30 12:01:49.000155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.000132    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca" pod "35f86870-543e-4547-a11b-8c2d401f75b6" (UID: "35f86870-543e-4547-a11b-8c2d401f75b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:49.000487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.000470    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:49.002192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.002160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb" pod "35f86870-543e-4547-a11b-8c2d401f75b6" (UID: "35f86870-543e-4547-a11b-8c2d401f75b6"). InnerVolumeSpecName "kube-api-access-qdcvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:49.002293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.002273    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca" pod "35f86870-543e-4547-a11b-8c2d401f75b6" (UID: "35f86870-543e-4547-a11b-8c2d401f75b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:01:49.102658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.102577    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdcvb\" (UniqueName: \"kubernetes.io/projected/35f86870-543e-4547-a11b-8c2d401f75b6-kube-api-access-qdcvb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:49.102658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.102610    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f86870-543e-4547-a11b-8c2d401f75b6-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:49.627736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.627703    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:49.789236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.789194    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:49.809668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.809628    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm" event={"ID":"35f86870-543e-4547-a11b-8c2d401f75b6","Type":"ContainerDied","Data":"2c097edd36343ab9720ebe676e9c761b9d295ff932f3ce7097507bf52e10a80d"}
Oct 30 12:01:49.810072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.809683    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c097edd36343ab9720ebe676e9c761b9d295ff932f3ce7097507bf52e10a80d"
Oct 30 12:01:49.810072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:49.809648    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"
Oct 30 12:01:49.859189 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:01:49.859156    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:01:50.057757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.057711    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"]
Oct 30 12:01:50.058679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.058655    5060 cpu_manager.go:414] "containerMap: removing container" podUID="35f86870-543e-4547-a11b-8c2d401f75b6" containerName="revoke-keys"
Oct 30 12:01:50.058814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.058680    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f86870-543e-4547-a11b-8c2d401f75b6" containerName="revoke-keys"
Oct 30 12:01:50.058814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.058750    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d190f090-23ff-46be-85f0-c001db92d9e7" containerName="revoke-keys"
Oct 30 12:01:50.058814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.058759    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d190f090-23ff-46be-85f0-c001db92d9e7" containerName="revoke-keys"
Oct 30 12:01:50.058971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.058935    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d190f090-23ff-46be-85f0-c001db92d9e7" containerName="revoke-keys"
Oct 30 12:01:50.059033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.058971    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f86870-543e-4547-a11b-8c2d401f75b6" containerName="revoke-keys"
Oct 30 12:01:50.063418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.063392    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.081039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.080998    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:01:50.081370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.081290    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:01:50.081507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.081434    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:01:50.084947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.081779    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:01:50.088399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.088049    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"]
Oct 30 12:01:50.115341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.115307    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.115605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.115581    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57mj\" (UniqueName: \"kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.115715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.115698    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.217187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.217150    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.217377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.217356    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57mj\" (UniqueName: \"kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.217470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.217439    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.218195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.218165    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.219817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.219797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.241360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.241327    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57mj\" (UniqueName: \"kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj\") pod \"maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.381622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.381533    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:50.500155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.500067    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:50.696771 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:01:50.696733    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c72a75_e6a0_4710_8556_9b293baa4b48.slice/crio-a2b980e7bfe2b2a26e6f8de2f211eecc31bef14040701714e22ddcf692a81a83 WatchSource:0}: Error finding container a2b980e7bfe2b2a26e6f8de2f211eecc31bef14040701714e22ddcf692a81a83: Status 404 returned error can't find the container with id a2b980e7bfe2b2a26e6f8de2f211eecc31bef14040701714e22ddcf692a81a83
Oct 30 12:01:50.697918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.697881    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"]
Oct 30 12:01:50.770157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.770117    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:50.821168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:50.821128    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45" event={"ID":"f4c72a75-e6a0-4710-8556-9b293baa4b48","Type":"ContainerStarted","Data":"a2b980e7bfe2b2a26e6f8de2f211eecc31bef14040701714e22ddcf692a81a83"}
Oct 30 12:01:51.832310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:51.832264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45" event={"ID":"f4c72a75-e6a0-4710-8556-9b293baa4b48","Type":"ContainerStarted","Data":"99e40fa5ebe7fb374a0f3b0a601413c50f4c0ef80ef9d1f26c2ef9c60c196d8f"}
Oct 30 12:01:51.876193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:51.876130    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45" podStartSLOduration=1.876111101 podStartE2EDuration="1.876111101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:51.872745642 +0000 UTC m=+1915.667630136" watchObservedRunningTime="2026-10-30 12:01:51.876111101 +0000 UTC m=+1915.670995602"
Oct 30 12:01:52.582572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:52.582534    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:52.843130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:52.843042    5060 generic.go:356] "Generic (PLEG): container finished" podID="f4c72a75-e6a0-4710-8556-9b293baa4b48" containerID="99e40fa5ebe7fb374a0f3b0a601413c50f4c0ef80ef9d1f26c2ef9c60c196d8f" exitCode=0
Oct 30 12:01:52.843537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:52.843130    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45" event={"ID":"f4c72a75-e6a0-4710-8556-9b293baa4b48","Type":"ContainerDied","Data":"99e40fa5ebe7fb374a0f3b0a601413c50f4c0ef80ef9d1f26c2ef9c60c196d8f"}
Oct 30 12:01:54.005854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.005825    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:54.065377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.065345    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca\") pod \"f4c72a75-e6a0-4710-8556-9b293baa4b48\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") "
Oct 30 12:01:54.065622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.065543    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj\" (UniqueName: \"kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj\") pod \"f4c72a75-e6a0-4710-8556-9b293baa4b48\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") "
Oct 30 12:01:54.065622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.065595    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca\") pod \"f4c72a75-e6a0-4710-8556-9b293baa4b48\" (UID: \"f4c72a75-e6a0-4710-8556-9b293baa4b48\") "
Oct 30 12:01:54.065793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.065764    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca" pod "f4c72a75-e6a0-4710-8556-9b293baa4b48" (UID: "f4c72a75-e6a0-4710-8556-9b293baa4b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:01:54.066071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.066053    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:54.067814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.067783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca" pod "f4c72a75-e6a0-4710-8556-9b293baa4b48" (UID: "f4c72a75-e6a0-4710-8556-9b293baa4b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:01:54.068067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.068042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj" pod "f4c72a75-e6a0-4710-8556-9b293baa4b48" (UID: "f4c72a75-e6a0-4710-8556-9b293baa4b48"). InnerVolumeSpecName "kube-api-access-d57mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:01:54.167938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.167853    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c72a75-e6a0-4710-8556-9b293baa4b48-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:54.167938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.167887    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57mj\" (UniqueName: \"kubernetes.io/projected/f4c72a75-e6a0-4710-8556-9b293baa4b48-kube-api-access-d57mj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:01:54.460860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.460831    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:54.861486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.861385    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"
Oct 30 12:01:54.861486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.861383    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45" event={"ID":"f4c72a75-e6a0-4710-8556-9b293baa4b48","Type":"ContainerDied","Data":"a2b980e7bfe2b2a26e6f8de2f211eecc31bef14040701714e22ddcf692a81a83"}
Oct 30 12:01:54.861689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:54.861528    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2b980e7bfe2b2a26e6f8de2f211eecc31bef14040701714e22ddcf692a81a83"
Oct 30 12:01:56.386253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:56.386219    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:57.772480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:57.772427    5060 scope.go:122] "RemoveContainer" containerID="aa5deea568d44c8c0c4cfdc015991f3b3c73ec0fd733f8b1d00ea014f19cf439"
Oct 30 12:01:58.161250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:58.161160    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:01:59.949491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:01:59.949418    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:00.535806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:00.535759    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"]
Oct 30 12:02:00.571658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:00.571620    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"]
Oct 30 12:02:00.707627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:00.707592    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:00.877305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:00.877208    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:01.841210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:01.841171    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:03.640909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:03.640869    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:04.321224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:04.321182    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:04.466230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:04.466172    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:05.529477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.528861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:05.538479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.536508    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf" podUID="1d047d9e-ebb3-4d19-97ed-0982641356e0" containerName="payload-processing" containerID="cri-o://42a38d7a62df913f9fae598982cefa839fee609946dcadb4a3cc653e207945fe" gracePeriod=25
Oct 30 12:02:05.572859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.572820    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q" podUID="742cb5a2-d059-4fe4-a742-2e84248366d9" containerName="payload-pre-processing" containerID="cri-o://882db9eb24bc62d653c7e95882ef4fa0fffe7cf97d45dff3d837fe96ec80c539" gracePeriod=25
Oct 30 12:02:05.668093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.668051    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"]
Oct 30 12:02:05.668807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.668786    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f4c72a75-e6a0-4710-8556-9b293baa4b48" containerName="revoke-keys"
Oct 30 12:02:05.668807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.668806    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c72a75-e6a0-4710-8556-9b293baa4b48" containerName="revoke-keys"
Oct 30 12:02:05.668948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.668925    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c72a75-e6a0-4710-8556-9b293baa4b48" containerName="revoke-keys"
Oct 30 12:02:05.672618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.672584    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.751009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.750972    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"]
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848212    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz65\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848287    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848326    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848382    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848429    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848487    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.849073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.848606    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.949892    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.949962    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950072    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950112    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950137    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz65\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950197    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950253    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.950479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950348    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.951171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950627    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.951171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.950873    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.951171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.951067    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.951171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.951102    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.951421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.951335    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.954054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.954029    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.954225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.954208    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:05.966713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.966657    5060 generic.go:356] "Generic (PLEG): container finished" podID="1d047d9e-ebb3-4d19-97ed-0982641356e0" containerID="42a38d7a62df913f9fae598982cefa839fee609946dcadb4a3cc653e207945fe" exitCode=0
Oct 30 12:02:05.967083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.967039    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf" event={"ID":"1d047d9e-ebb3-4d19-97ed-0982641356e0","Type":"ContainerDied","Data":"42a38d7a62df913f9fae598982cefa839fee609946dcadb4a3cc653e207945fe"}
Oct 30 12:02:05.972680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.971033    5060 generic.go:356] "Generic (PLEG): container finished" podID="742cb5a2-d059-4fe4-a742-2e84248366d9" containerID="882db9eb24bc62d653c7e95882ef4fa0fffe7cf97d45dff3d837fe96ec80c539" exitCode=0
Oct 30 12:02:05.972680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:05.971154    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q" event={"ID":"742cb5a2-d059-4fe4-a742-2e84248366d9","Type":"ContainerDied","Data":"882db9eb24bc62d653c7e95882ef4fa0fffe7cf97d45dff3d837fe96ec80c539"}
Oct 30 12:02:06.033479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.033176    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:06.043508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.042172    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz65\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:06.048044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.047727    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:06.120862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.120535    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 12:02:06.167618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.167581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume\") pod \"1d047d9e-ebb3-4d19-97ed-0982641356e0\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") "
Oct 30 12:02:06.168378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.167812    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x\" (UniqueName: \"kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x\") pod \"1d047d9e-ebb3-4d19-97ed-0982641356e0\" (UID: \"1d047d9e-ebb3-4d19-97ed-0982641356e0\") "
Oct 30 12:02:06.168378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.168086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume" pod "1d047d9e-ebb3-4d19-97ed-0982641356e0" (UID: "1d047d9e-ebb3-4d19-97ed-0982641356e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:06.168378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.168349    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d047d9e-ebb3-4d19-97ed-0982641356e0-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:06.183697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.183651    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x" pod "1d047d9e-ebb3-4d19-97ed-0982641356e0" (UID: "1d047d9e-ebb3-4d19-97ed-0982641356e0"). InnerVolumeSpecName "kube-api-access-8mb4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:06.203295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.203010    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 12:02:06.269446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.269218    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km\" (UniqueName: \"kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km\") pod \"742cb5a2-d059-4fe4-a742-2e84248366d9\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") "
Oct 30 12:02:06.269446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.269289    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume\") pod \"742cb5a2-d059-4fe4-a742-2e84248366d9\" (UID: \"742cb5a2-d059-4fe4-a742-2e84248366d9\") "
Oct 30 12:02:06.269924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.269898    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb4x\" (UniqueName: \"kubernetes.io/projected/1d047d9e-ebb3-4d19-97ed-0982641356e0-kube-api-access-8mb4x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:06.270031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.269957    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume" pod "742cb5a2-d059-4fe4-a742-2e84248366d9" (UID: "742cb5a2-d059-4fe4-a742-2e84248366d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:06.272597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.272555    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km" pod "742cb5a2-d059-4fe4-a742-2e84248366d9" (UID: "742cb5a2-d059-4fe4-a742-2e84248366d9"). InnerVolumeSpecName "kube-api-access-jg2km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:06.371236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.371127    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2km\" (UniqueName: \"kubernetes.io/projected/742cb5a2-d059-4fe4-a742-2e84248366d9-kube-api-access-jg2km\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:06.371236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.371161    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742cb5a2-d059-4fe4-a742-2e84248366d9-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:06.424278 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:06.424243    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bae7157_5ca6_42fc_9a4c_659843476660.slice/crio-7898d57ebec26107b00f1a0d8c7c36f0ea301f02b2f3813f81315447988f3126 WatchSource:0}: Error finding container 7898d57ebec26107b00f1a0d8c7c36f0ea301f02b2f3813f81315447988f3126: Status 404 returned error can't find the container with id 7898d57ebec26107b00f1a0d8c7c36f0ea301f02b2f3813f81315447988f3126
Oct 30 12:02:06.424833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.424807    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"]
Oct 30 12:02:06.426797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.426765    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:02:06.426907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.426845    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:02:06.426907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.426890    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:02:06.981405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.981359    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" event={"ID":"9bae7157-5ca6-42fc-9a4c-659843476660","Type":"ContainerStarted","Data":"f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075"}
Oct 30 12:02:06.981964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.981427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" event={"ID":"9bae7157-5ca6-42fc-9a4c-659843476660","Type":"ContainerStarted","Data":"7898d57ebec26107b00f1a0d8c7c36f0ea301f02b2f3813f81315447988f3126"}
Oct 30 12:02:06.983149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.983128    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"
Oct 30 12:02:06.983276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.983168    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf" event={"ID":"1d047d9e-ebb3-4d19-97ed-0982641356e0","Type":"ContainerDied","Data":"4e8f39f2d541d66f58684a41b3a4baf255e3b068de3398e63a2e39cb8d57f338"}
Oct 30 12:02:06.983276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.983226    5060 scope.go:122] "RemoveContainer" containerID="42a38d7a62df913f9fae598982cefa839fee609946dcadb4a3cc653e207945fe"
Oct 30 12:02:06.984929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.984874    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q" event={"ID":"742cb5a2-d059-4fe4-a742-2e84248366d9","Type":"ContainerDied","Data":"cc84855721891fc5506bb0a5b40a4b2c10d48a045aeeec4a3c2734411ebc07f3"}
Oct 30 12:02:06.984929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.984884    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"
Oct 30 12:02:06.996733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:06.996711    5060 scope.go:122] "RemoveContainer" containerID="882db9eb24bc62d653c7e95882ef4fa0fffe7cf97d45dff3d837fe96ec80c539"
Oct 30 12:02:07.030640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.030580    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podStartSLOduration=2.030566111 podStartE2EDuration="2.030566111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:02:07.026815813 +0000 UTC m=+1930.821700306" watchObservedRunningTime="2026-10-30 12:02:07.030566111 +0000 UTC m=+1930.825450604"
Oct 30 12:02:07.048698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.048658    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:07.050233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.050207    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:07.050382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.050254    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:07.066726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.066693    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"]
Oct 30 12:02:07.070691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.070666    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4093c5-6dfcd5c987-8lm4q"]
Oct 30 12:02:07.106523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.104325    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"]
Oct 30 12:02:07.108079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.108051    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4093c5-76f9565568-t5dcf"]
Oct 30 12:02:07.429104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:07.428997    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:08.049222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.049189    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:08.049682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.049273    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:08.115319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.115273    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"]
Oct 30 12:02:08.116225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.116201    5060 cpu_manager.go:414] "containerMap: removing container" podUID="742cb5a2-d059-4fe4-a742-2e84248366d9" containerName="payload-pre-processing"
Oct 30 12:02:08.116304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.116230    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742cb5a2-d059-4fe4-a742-2e84248366d9" containerName="payload-pre-processing"
Oct 30 12:02:08.116349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.116306    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1d047d9e-ebb3-4d19-97ed-0982641356e0" containerName="payload-processing"
Oct 30 12:02:08.116349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.116315    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d047d9e-ebb3-4d19-97ed-0982641356e0" containerName="payload-processing"
Oct 30 12:02:08.116508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.116497    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d047d9e-ebb3-4d19-97ed-0982641356e0" containerName="payload-processing"
Oct 30 12:02:08.116566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.116514    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="742cb5a2-d059-4fe4-a742-2e84248366d9" containerName="payload-pre-processing"
Oct 30 12:02:08.121735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.121711    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.134178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.133800    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"]
Oct 30 12:02:08.142146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.142083    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:02:08.142388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.142367    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:02:08.142479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.142374    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:02:08.142479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.142440    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:02:08.218972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.218937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.219207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.219192    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.219430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.219417    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7rk\" (UniqueName: \"kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.320550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.320427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.320550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.320493    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.320834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.320645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7rk\" (UniqueName: \"kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.321275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.321247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.323227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.323199    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.336256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.336222    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d047d9e-ebb3-4d19-97ed-0982641356e0" path="/var/lib/kubelet/pods/1d047d9e-ebb3-4d19-97ed-0982641356e0/volumes"
Oct 30 12:02:08.336711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.336696    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742cb5a2-d059-4fe4-a742-2e84248366d9" path="/var/lib/kubelet/pods/742cb5a2-d059-4fe4-a742-2e84248366d9/volumes"
Oct 30 12:02:08.345017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.344989    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7rk\" (UniqueName: \"kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk\") pod \"maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.438430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.438393    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:08.774706 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:08.774667    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba985e27_7081_4e92_9e75_19242b6b7481.slice/crio-39369f56312c91d27a568798580e57bdcabb6910f8dde5601479b896ba715c17 WatchSource:0}: Error finding container 39369f56312c91d27a568798580e57bdcabb6910f8dde5601479b896ba715c17: Status 404 returned error can't find the container with id 39369f56312c91d27a568798580e57bdcabb6910f8dde5601479b896ba715c17
Oct 30 12:02:08.776028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:08.776004    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"]
Oct 30 12:02:09.029538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:09.028732    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8" event={"ID":"ba985e27-7081-4e92-9e75-19242b6b7481","Type":"ContainerStarted","Data":"39369f56312c91d27a568798580e57bdcabb6910f8dde5601479b896ba715c17"}
Oct 30 12:02:09.049334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:09.049298    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:09.049808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:09.049351    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:09.521245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:09.521179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:10.038345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:10.038305    5060 generic.go:356] "Generic (PLEG): container finished" podID="ba985e27-7081-4e92-9e75-19242b6b7481" containerID="7572704dcf2d0e0b0374302f56e7350637fa9071553c47e679fcb79ed0973181" exitCode=0
Oct 30 12:02:10.038603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:10.038388    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8" event={"ID":"ba985e27-7081-4e92-9e75-19242b6b7481","Type":"ContainerDied","Data":"7572704dcf2d0e0b0374302f56e7350637fa9071553c47e679fcb79ed0973181"}
Oct 30 12:02:10.049008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:10.048978    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:10.049145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:10.049028    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:11.049051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.049023    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:11.049440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.049081    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:11.259275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.259246    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:11.364274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.364187    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca\") pod \"ba985e27-7081-4e92-9e75-19242b6b7481\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") "
Oct 30 12:02:11.364433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.364383    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca\") pod \"ba985e27-7081-4e92-9e75-19242b6b7481\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") "
Oct 30 12:02:11.364433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.364423    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk\" (UniqueName: \"kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk\") pod \"ba985e27-7081-4e92-9e75-19242b6b7481\" (UID: \"ba985e27-7081-4e92-9e75-19242b6b7481\") "
Oct 30 12:02:11.364826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.364789    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca" pod "ba985e27-7081-4e92-9e75-19242b6b7481" (UID: "ba985e27-7081-4e92-9e75-19242b6b7481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:11.366556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.366529    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca" pod "ba985e27-7081-4e92-9e75-19242b6b7481" (UID: "ba985e27-7081-4e92-9e75-19242b6b7481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:02:11.366746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.366728    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk" pod "ba985e27-7081-4e92-9e75-19242b6b7481" (UID: "ba985e27-7081-4e92-9e75-19242b6b7481"). InnerVolumeSpecName "kube-api-access-2w7rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:11.466351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.466319    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:11.466351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.466349    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7rk\" (UniqueName: \"kubernetes.io/projected/ba985e27-7081-4e92-9e75-19242b6b7481-kube-api-access-2w7rk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:11.466351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:11.466359    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba985e27-7081-4e92-9e75-19242b6b7481-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:12.048551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:12.048518    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:12.048748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:12.048575    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:12.054990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:12.054961    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:12.057867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:12.057847    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"
Oct 30 12:02:12.058011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:12.057863    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8" event={"ID":"ba985e27-7081-4e92-9e75-19242b6b7481","Type":"ContainerDied","Data":"39369f56312c91d27a568798580e57bdcabb6910f8dde5601479b896ba715c17"}
Oct 30 12:02:12.058011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:12.057931    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39369f56312c91d27a568798580e57bdcabb6910f8dde5601479b896ba715c17"
Oct 30 12:02:13.049397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:13.049360    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 12:02:13.049651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:13.049414    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 12:02:14.053319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.053288    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:14.053861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.053783    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:14.054533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.054510    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:02:14.193242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.193208    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:14.817389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.817343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:14.917212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.917181    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"]
Oct 30 12:02:14.929325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.929296    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4093c5-vww45"]
Oct 30 12:02:14.967019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:14.966948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:15.443388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:15.443356    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:16.346025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:16.345985    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c72a75-e6a0-4710-8556-9b293baa4b48" path="/var/lib/kubelet/pods/f4c72a75-e6a0-4710-8556-9b293baa4b48/volumes"
Oct 30 12:02:16.844957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:16.844915    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:18.126483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.126426    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"]
Oct 30 12:02:18.127016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.127001    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ba985e27-7081-4e92-9e75-19242b6b7481" containerName="revoke-keys"
Oct 30 12:02:18.127095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.127017    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba985e27-7081-4e92-9e75-19242b6b7481" containerName="revoke-keys"
Oct 30 12:02:18.127151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.127138    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba985e27-7081-4e92-9e75-19242b6b7481" containerName="revoke-keys"
Oct 30 12:02:18.130582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.130560    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.148608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.148566    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"]
Oct 30 12:02:18.155789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.155756    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:02:18.156092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.156065    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:02:18.156231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.156213    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:02:18.156317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.155994    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:02:18.279998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.279966    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.279998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.280012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxhh\" (UniqueName: \"kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.280215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.280155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.381222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.381130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.381222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.381173    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxhh\" (UniqueName: \"kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.381537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.381248    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.382104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.382079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.383856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.383833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.405775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.405737    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxhh\" (UniqueName: \"kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh\") pod \"maas-api-revoke-sub-a40b7a39-99x2v\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.444884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.444849    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:18.641909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.641804    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:18.754309 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:18.754272    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3759011_dbd0_4ff6_91ac_d7aa12cbc0b4.slice/crio-1d0f2b8ad059f9c4a0c52c03da85e19d1559f537504d9d600cf4e7561b575c78 WatchSource:0}: Error finding container 1d0f2b8ad059f9c4a0c52c03da85e19d1559f537504d9d600cf4e7561b575c78: Status 404 returned error can't find the container with id 1d0f2b8ad059f9c4a0c52c03da85e19d1559f537504d9d600cf4e7561b575c78
Oct 30 12:02:18.755408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:18.755386    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"]
Oct 30 12:02:19.118867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:19.118835    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v" event={"ID":"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4","Type":"ContainerStarted","Data":"1d0f2b8ad059f9c4a0c52c03da85e19d1559f537504d9d600cf4e7561b575c78"}
Oct 30 12:02:19.858687 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:02:19.858649    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:02:20.138136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:20.138053    5060 generic.go:356] "Generic (PLEG): container finished" podID="d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" containerID="eb4a6c089e94ef3dd5d72af105c90aa7480489d45e433d37b3c8ec7944fce4de" exitCode=0
Oct 30 12:02:20.138294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:20.138139    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v" event={"ID":"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4","Type":"ContainerDied","Data":"eb4a6c089e94ef3dd5d72af105c90aa7480489d45e433d37b3c8ec7944fce4de"}
Oct 30 12:02:20.324143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:20.324102    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:21.316018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.315988    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:21.421490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.421367    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca\") pod \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") "
Oct 30 12:02:21.421687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.421555    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca\") pod \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") "
Oct 30 12:02:21.421687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.421654    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh\" (UniqueName: \"kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh\") pod \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\" (UID: \"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4\") "
Oct 30 12:02:21.421926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.421901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca" pod "d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" (UID: "d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:21.422166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.422151    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:21.423828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.423801    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca" pod "d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" (UID: "d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:02:21.423953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.423890    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh" pod "d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" (UID: "d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4"). InnerVolumeSpecName "kube-api-access-hsxhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:21.472724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.472680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:21.523839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.523801    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxhh\" (UniqueName: \"kubernetes.io/projected/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-kube-api-access-hsxhh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:21.523839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.523833    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:21.630598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.630561    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:21.759466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.759409    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"]
Oct 30 12:02:21.759747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.759718    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy" containerID="cri-o://b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378" gracePeriod=30
Oct 30 12:02:21.821009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:21.820973    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:22.154979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.154888    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"
Oct 30 12:02:22.154979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.154928    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v" event={"ID":"d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4","Type":"ContainerDied","Data":"1d0f2b8ad059f9c4a0c52c03da85e19d1559f537504d9d600cf4e7561b575c78"}
Oct 30 12:02:22.154979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.154974    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d0f2b8ad059f9c4a0c52c03da85e19d1559f537504d9d600cf4e7561b575c78"
Oct 30 12:02:22.268338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.268299    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:22.367934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.367900    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"]
Oct 30 12:02:22.368510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.368475    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" containerName="revoke-keys"
Oct 30 12:02:22.368510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.368492    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" containerName="revoke-keys"
Oct 30 12:02:22.368652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.368639    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" containerName="revoke-keys"
Oct 30 12:02:22.372150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.372129    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.388300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.388273    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:02:22.388477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.388324    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:02:22.390906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.390882    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"]
Oct 30 12:02:22.430094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.430004    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:02:22.430346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.430147    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:02:22.437737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.437707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.437877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.437858    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdlrc\" (UniqueName: \"kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.437964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.437948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.539829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.539790    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.540050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.539887    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdlrc\" (UniqueName: \"kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.540050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.539986    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.540516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.540446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.542645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.542618    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.563934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.563905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdlrc\" (UniqueName: \"kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc\") pod \"maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.688409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.688309    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:22.995018 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:22.994984    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750f842b_52ad_4e80_939f_236e12d7f44d.slice/crio-2d4df7f352bca0493ca256b39e239d2d3677a9b3db19dd64eaedf5d394c8bc38 WatchSource:0}: Error finding container 2d4df7f352bca0493ca256b39e239d2d3677a9b3db19dd64eaedf5d394c8bc38: Status 404 returned error can't find the container with id 2d4df7f352bca0493ca256b39e239d2d3677a9b3db19dd64eaedf5d394c8bc38
Oct 30 12:02:22.996130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:22.996103    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"]
Oct 30 12:02:23.164193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:23.164160    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v" event={"ID":"750f842b-52ad-4e80-939f-236e12d7f44d","Type":"ContainerStarted","Data":"2d4df7f352bca0493ca256b39e239d2d3677a9b3db19dd64eaedf5d394c8bc38"}
Oct 30 12:02:24.174020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:24.173984    5060 generic.go:356] "Generic (PLEG): container finished" podID="750f842b-52ad-4e80-939f-236e12d7f44d" containerID="2b9e92b4d1b5e4e4b0d3098c91caa4e11a07f1c818336207b0918680523d51ea" exitCode=0
Oct 30 12:02:24.174440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:24.174080    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v" event={"ID":"750f842b-52ad-4e80-939f-236e12d7f44d","Type":"ContainerDied","Data":"2b9e92b4d1b5e4e4b0d3098c91caa4e11a07f1c818336207b0918680523d51ea"}
Oct 30 12:02:24.204847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:24.204810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:25.366256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.366228    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:25.477167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.477120    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc\" (UniqueName: \"kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc\") pod \"750f842b-52ad-4e80-939f-236e12d7f44d\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") "
Oct 30 12:02:25.477391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.477211    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca\") pod \"750f842b-52ad-4e80-939f-236e12d7f44d\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") "
Oct 30 12:02:25.477391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.477265    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca\") pod \"750f842b-52ad-4e80-939f-236e12d7f44d\" (UID: \"750f842b-52ad-4e80-939f-236e12d7f44d\") "
Oct 30 12:02:25.477726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.477695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca" pod "750f842b-52ad-4e80-939f-236e12d7f44d" (UID: "750f842b-52ad-4e80-939f-236e12d7f44d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:25.477913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.477888    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:25.479659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.479634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca" pod "750f842b-52ad-4e80-939f-236e12d7f44d" (UID: "750f842b-52ad-4e80-939f-236e12d7f44d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:02:25.480018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.479993    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc" pod "750f842b-52ad-4e80-939f-236e12d7f44d" (UID: "750f842b-52ad-4e80-939f-236e12d7f44d"). InnerVolumeSpecName "kube-api-access-fdlrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:25.579143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.579103    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750f842b-52ad-4e80-939f-236e12d7f44d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:25.579143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.579135    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdlrc\" (UniqueName: \"kubernetes.io/projected/750f842b-52ad-4e80-939f-236e12d7f44d-kube-api-access-fdlrc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:25.975134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:25.975094    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:26.195464 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:26.195417    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"
Oct 30 12:02:26.195652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:26.195469    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v" event={"ID":"750f842b-52ad-4e80-939f-236e12d7f44d","Type":"ContainerDied","Data":"2d4df7f352bca0493ca256b39e239d2d3677a9b3db19dd64eaedf5d394c8bc38"}
Oct 30 12:02:26.195652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:26.195559    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4df7f352bca0493ca256b39e239d2d3677a9b3db19dd64eaedf5d394c8bc38"
Oct 30 12:02:27.067132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.067101    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 12:02:27.203904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.203865    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.203979    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204076    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204223    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204261    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204374    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204501    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.204600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.204539    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket\") pod \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\" (UID: \"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea\") "
Oct 30 12:02:27.205115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.205083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:27.205531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.205255    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:27.206746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.206097    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:27.206746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.206404    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:27.206746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.206699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:27.206746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.206726    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:02:27.208099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.208071    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "kube-api-access-5q2wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:27.208220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.208162    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:27.209910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.209872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token" pod "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" (UID: "8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:27.210595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.210568    5060 generic.go:356] "Generic (PLEG): container finished" podID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerID="b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378" exitCode=0
Oct 30 12:02:27.210705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.210635    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"
Oct 30 12:02:27.211068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.210813    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" event={"ID":"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea","Type":"ContainerDied","Data":"b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378"}
Oct 30 12:02:27.211068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.210891    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl" event={"ID":"8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea","Type":"ContainerDied","Data":"ebd116c488a1f51d3869c77a2f502e0921e271ff96e8285fbefb3cb5b01eb430"}
Oct 30 12:02:27.211068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.210928    5060 scope.go:122] "RemoveContainer" containerID="b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378"
Oct 30 12:02:27.243672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.241869    5060 scope.go:122] "RemoveContainer" containerID="beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"
Oct 30 12:02:27.256685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.256663    5060 scope.go:122] "RemoveContainer" containerID="b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378"
Oct 30 12:02:27.256997 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:02:27.256979    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378\": container with ID starting with b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378 not found: ID does not exist" containerID="b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378"
Oct 30 12:02:27.257075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.257009    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378"} err="failed to get container status \"b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378\": rpc error: code = NotFound desc = could not find container \"b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378\": container with ID starting with b8cd9fc0b344ec60a9420f443649f4f73a70f99d4a9c006b5c18fb28577ac378 not found: ID does not exist"
Oct 30 12:02:27.257075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.257033    5060 scope.go:122] "RemoveContainer" containerID="beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"
Oct 30 12:02:27.257286 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:02:27.257267    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47\": container with ID starting with beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47 not found: ID does not exist" containerID="beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"
Oct 30 12:02:27.257326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.257293    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47"} err="failed to get container status \"beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47\": rpc error: code = NotFound desc = could not find container \"beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47\": container with ID starting with beb6b8d3f8f3e59679cef1e5a785d27517728864696438cd20018ae6c4f93a47 not found: ID does not exist"
Oct 30 12:02:27.275022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.274958    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"]
Oct 30 12:02:27.282166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.282125    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4093c5-openshift-default-79d57c889c-cj7hl"]
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306365    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306399    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306413    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306424    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306435    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306446    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306482    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306495    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2wr\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-kube-api-access-5q2wr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.306891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.306507    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:27.768122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:27.768086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:28.335594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.335559    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" path="/var/lib/kubelet/pods/8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea/volumes"
Oct 30 12:02:28.491152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.491106    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"]
Oct 30 12:02:28.491961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.491939    5060 cpu_manager.go:414] "containerMap: removing container" podUID="750f842b-52ad-4e80-939f-236e12d7f44d" containerName="revoke-keys"
Oct 30 12:02:28.492019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.491965    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750f842b-52ad-4e80-939f-236e12d7f44d" containerName="revoke-keys"
Oct 30 12:02:28.492019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492002    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy"
Oct 30 12:02:28.492019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492009    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy"
Oct 30 12:02:28.492118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492043    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy"
Oct 30 12:02:28.492118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492050    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy"
Oct 30 12:02:28.492188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492177    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy"
Oct 30 12:02:28.492228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492193    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1a886e-4f3a-4d6b-a51a-5238a17bb9ea" containerName="istio-proxy"
Oct 30 12:02:28.492228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.492207    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="750f842b-52ad-4e80-939f-236e12d7f44d" containerName="revoke-keys"
Oct 30 12:02:28.497928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.497902    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.515908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.515874    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"]
Oct 30 12:02:28.521765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.521737    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:02:28.523931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.522047    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:02:28.524124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.522137    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:02:28.525289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.525266    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:02:28.624011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.623920    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.624220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.624120    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5ktn\" (UniqueName: \"kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.624220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.624204    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.725485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.725401    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.725673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.725612    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ktn\" (UniqueName: \"kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.725717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.725678    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.726147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.726114    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.728393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.728367    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.750211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.750179    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ktn\" (UniqueName: \"kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn\") pod \"maas-api-revoke-sub-b662bb8b-msb2z\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:28.815313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:28.815268    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:29.130705 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:29.130667    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07da6c7c_0bb6_4668_9ec9_14ca7c181d7c.slice/crio-ac301a313c12df21382cb9bd0c2764914c8a8b9c5de00d129a20ee8ce7653bee WatchSource:0}: Error finding container ac301a313c12df21382cb9bd0c2764914c8a8b9c5de00d129a20ee8ce7653bee: Status 404 returned error can't find the container with id ac301a313c12df21382cb9bd0c2764914c8a8b9c5de00d129a20ee8ce7653bee
Oct 30 12:02:29.131730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:29.131708    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"]
Oct 30 12:02:29.232709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:29.232671    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z" event={"ID":"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c","Type":"ContainerStarted","Data":"ac301a313c12df21382cb9bd0c2764914c8a8b9c5de00d129a20ee8ce7653bee"}
Oct 30 12:02:29.459288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:29.459258    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:30.247664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:30.247614    5060 generic.go:356] "Generic (PLEG): container finished" podID="07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" containerID="858fc764d1433834776e3db23df2d9547c640cb3738a749da0b0140e5812d8aa" exitCode=0
Oct 30 12:02:30.247909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:30.247884    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z" event={"ID":"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c","Type":"ContainerDied","Data":"858fc764d1433834776e3db23df2d9547c640cb3738a749da0b0140e5812d8aa"}
Oct 30 12:02:31.439293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.439256    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:31.477764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.477674    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca\") pod \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") "
Oct 30 12:02:31.477764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.477741    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca\") pod \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") "
Oct 30 12:02:31.478129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.477817    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn\" (UniqueName: \"kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn\") pod \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\" (UID: \"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c\") "
Oct 30 12:02:31.478529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.478493    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca" pod "07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" (UID: "07da6c7c-0bb6-4668-9ec9-14ca7c181d7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:31.481144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.481116    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca" pod "07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" (UID: "07da6c7c-0bb6-4668-9ec9-14ca7c181d7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:02:31.488025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.487993    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn" pod "07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" (UID: "07da6c7c-0bb6-4668-9ec9-14ca7c181d7c"). InnerVolumeSpecName "kube-api-access-n5ktn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:31.579327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.579239    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:31.579327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.579277    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:31.579327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.579293    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5ktn\" (UniqueName: \"kubernetes.io/projected/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c-kube-api-access-n5ktn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:31.781690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:31.781637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:32.265617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:32.265583    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z" event={"ID":"07da6c7c-0bb6-4668-9ec9-14ca7c181d7c","Type":"ContainerDied","Data":"ac301a313c12df21382cb9bd0c2764914c8a8b9c5de00d129a20ee8ce7653bee"}
Oct 30 12:02:32.265617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:32.265606    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"
Oct 30 12:02:32.265902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:32.265631    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac301a313c12df21382cb9bd0c2764914c8a8b9c5de00d129a20ee8ce7653bee"
Oct 30 12:02:33.014482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.014407    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:33.156869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.156829    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:33.834429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.834396    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"]
Oct 30 12:02:33.834963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.834950    5060 cpu_manager.go:414] "containerMap: removing container" podUID="07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" containerName="revoke-keys"
Oct 30 12:02:33.835033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.834966    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" containerName="revoke-keys"
Oct 30 12:02:33.835081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.835072    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" containerName="revoke-keys"
Oct 30 12:02:33.838564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.838546    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:33.855803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.855768    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"]
Oct 30 12:02:33.863679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.863649    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:02:33.865726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.865695    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:02:33.865951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.865935    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:02:33.866097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.865948    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:02:33.912500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.912446    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:33.912795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.912774    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvvx\" (UniqueName: \"kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:33.913035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.913013    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:33.963382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:33.963345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:34.014310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.014266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.014759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.014736    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.015244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.015225    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvvx\" (UniqueName: \"kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.016064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.016038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.018011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.017965    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.040039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.040002    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvvx\" (UniqueName: \"kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx\") pod \"maas-api-revoke-sub-1e124709-lphr8\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.056148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.056113    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"]
Oct 30 12:02:34.056357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.056315    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz" podUID="1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" containerName="payload-processing" containerID="cri-o://a189a5cbeaaa53353ef5864deb2434fb7426c45457b70376e69da352edfa0d3f" gracePeriod=30
Oct 30 12:02:34.152496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.152313    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:34.170122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.170085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:34.288868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.288803    5060 generic.go:356] "Generic (PLEG): container finished" podID="1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" containerID="a189a5cbeaaa53353ef5864deb2434fb7426c45457b70376e69da352edfa0d3f" exitCode=0
Oct 30 12:02:34.289076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.289044    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz" event={"ID":"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123","Type":"ContainerDied","Data":"a189a5cbeaaa53353ef5864deb2434fb7426c45457b70376e69da352edfa0d3f"}
Oct 30 12:02:34.317532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.317414    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:34.517179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.517138    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 12:02:34.545668 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:34.545636    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40000dd_ee57_4ec6_9b9b_2b1d7b68f7a6.slice/crio-be983f177eeaea9b47119e348785599c659dcecc4722449bbfd988198dd2025e WatchSource:0}: Error finding container be983f177eeaea9b47119e348785599c659dcecc4722449bbfd988198dd2025e: Status 404 returned error can't find the container with id be983f177eeaea9b47119e348785599c659dcecc4722449bbfd988198dd2025e
Oct 30 12:02:34.547046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.547022    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"]
Oct 30 12:02:34.572275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.572225    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:34.640086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.640052    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt\" (UniqueName: \"kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt\") pod \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") "
Oct 30 12:02:34.640306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.640144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume\") pod \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\" (UID: \"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123\") "
Oct 30 12:02:34.640580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.640557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume" pod "1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" (UID: "1d6500d9-1e72-4ea3-a3fb-b51aed8c8123"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:34.642522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.642495    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt" pod "1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" (UID: "1d6500d9-1e72-4ea3-a3fb-b51aed8c8123"). InnerVolumeSpecName "kube-api-access-xkmkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:34.696013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.695978    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"]
Oct 30 12:02:34.701143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.701109    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-48e1f1-nbgf8"]
Oct 30 12:02:34.720219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.720117    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:34.741542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.741500    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmkt\" (UniqueName: \"kubernetes.io/projected/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-kube-api-access-xkmkt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:34.741542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:34.741531    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:35.246516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.246474    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:35.299682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.299579    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8" event={"ID":"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6","Type":"ContainerStarted","Data":"91ab0903b903a33eca75a4c736e092a75f40ee389e1968a32bff4aa9a01b7053"}
Oct 30 12:02:35.299682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.299644    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8" event={"ID":"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6","Type":"ContainerStarted","Data":"be983f177eeaea9b47119e348785599c659dcecc4722449bbfd988198dd2025e"}
Oct 30 12:02:35.301340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.301310    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz" event={"ID":"1d6500d9-1e72-4ea3-a3fb-b51aed8c8123","Type":"ContainerDied","Data":"3f050e05af498eb8ccc20e8aab53806bd6388152ad0594018653a01a6f7d479a"}
Oct 30 12:02:35.301991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.301351    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"
Oct 30 12:02:35.301991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.301365    5060 scope.go:122] "RemoveContainer" containerID="a189a5cbeaaa53353ef5864deb2434fb7426c45457b70376e69da352edfa0d3f"
Oct 30 12:02:35.348934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.348882    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8" podStartSLOduration=2.348867721 podStartE2EDuration="2.348867721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:02:35.345162188 +0000 UTC m=+1959.140046683" watchObservedRunningTime="2026-10-30 12:02:35.348867721 +0000 UTC m=+1959.143752215"
Oct 30 12:02:35.385913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.385869    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"]
Oct 30 12:02:35.392541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:35.392505    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-48e1f1-5bf98f5c6d-6rkmz"]
Oct 30 12:02:36.313229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:36.313190    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" containerID="91ab0903b903a33eca75a4c736e092a75f40ee389e1968a32bff4aa9a01b7053" exitCode=0
Oct 30 12:02:36.313732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:36.313246    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8" event={"ID":"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6","Type":"ContainerDied","Data":"91ab0903b903a33eca75a4c736e092a75f40ee389e1968a32bff4aa9a01b7053"}
Oct 30 12:02:36.335421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:36.335384    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" path="/var/lib/kubelet/pods/1d6500d9-1e72-4ea3-a3fb-b51aed8c8123/volumes"
Oct 30 12:02:36.335883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:36.335868    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba985e27-7081-4e92-9e75-19242b6b7481" path="/var/lib/kubelet/pods/ba985e27-7081-4e92-9e75-19242b6b7481/volumes"
Oct 30 12:02:36.655735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:36.655638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:37.599007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.598932    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:37.791743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.791710    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca\") pod \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") "
Oct 30 12:02:37.791945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.791879    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca\") pod \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") "
Oct 30 12:02:37.792016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.791971    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx\" (UniqueName: \"kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx\") pod \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\" (UID: \"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6\") "
Oct 30 12:02:37.792320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.792286    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca" pod "b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" (UID: "b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:37.794245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.794216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca" pod "b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" (UID: "b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:02:37.794377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.794289    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx" pod "b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" (UID: "b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6"). InnerVolumeSpecName "kube-api-access-xsvvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:37.893382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.893302    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsvvx\" (UniqueName: \"kubernetes.io/projected/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-kube-api-access-xsvvx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:37.893382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.893330    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:37.893382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:37.893341    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:38.335744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:38.335710    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"
Oct 30 12:02:38.339186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:38.339153    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8" event={"ID":"b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6","Type":"ContainerDied","Data":"be983f177eeaea9b47119e348785599c659dcecc4722449bbfd988198dd2025e"}
Oct 30 12:02:38.339338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:38.339212    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be983f177eeaea9b47119e348785599c659dcecc4722449bbfd988198dd2025e"
Oct 30 12:02:38.452499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:38.452442    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:39.961368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:39.961329    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:40.578779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:40.578740    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:40.697227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:40.697192    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"]
Oct 30 12:02:40.708681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:40.708643    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bebfd6-6ls4v"]
Oct 30 12:02:40.748676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:40.748644    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:40.898658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:40.898558    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:41.645998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:41.645962    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:41.791858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:41.791811    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:41.927992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:41.927893    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"]
Oct 30 12:02:41.928552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:41.928505    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" containerID="cri-o://a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4" gracePeriod=30
Oct 30 12:02:42.001296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.001265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:42.336672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.336640    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750f842b-52ad-4e80-939f-236e12d7f44d" path="/var/lib/kubelet/pods/750f842b-52ad-4e80-939f-236e12d7f44d/volumes"
Oct 30 12:02:42.542020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.541983    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"]
Oct 30 12:02:42.542532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.542519    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" containerName="payload-processing"
Oct 30 12:02:42.542578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.542533    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" containerName="payload-processing"
Oct 30 12:02:42.542578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.542570    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" containerName="revoke-keys"
Oct 30 12:02:42.542578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.542575    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" containerName="revoke-keys"
Oct 30 12:02:42.542690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.542681    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" containerName="revoke-keys"
Oct 30 12:02:42.542734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.542696    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6500d9-1e72-4ea3-a3fb-b51aed8c8123" containerName="payload-processing"
Oct 30 12:02:42.547500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.547481    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.569659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.569621    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.569818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.569780    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgmx\" (UniqueName: \"kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.569910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.569836    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.590689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.590581    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"]
Oct 30 12:02:42.596954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.596921    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:02:42.596954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.596921    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:02:42.597227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.596970    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:02:42.597227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.597127    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:02:42.670691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.670653    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgmx\" (UniqueName: \"kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.671199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.670718    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.671199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.670776    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.671566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.671544    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.673300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.673283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.712367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.712321    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgmx\" (UniqueName: \"kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:42.774177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.774137    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:42.861236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:42.861149    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:43.175720 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:02:43.175690    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8b12b2_6f87_4f90_b1ec_ef94c251aa69.slice/crio-c7ea5cbd052ca66ce6d91c91b098f02ff768c030d389aa3c279c3e4887b52e05 WatchSource:0}: Error finding container c7ea5cbd052ca66ce6d91c91b098f02ff768c030d389aa3c279c3e4887b52e05: Status 404 returned error can't find the container with id c7ea5cbd052ca66ce6d91c91b098f02ff768c030d389aa3c279c3e4887b52e05
Oct 30 12:02:43.177107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:43.176787    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"]
Oct 30 12:02:43.384220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:43.384184    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8" event={"ID":"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69","Type":"ContainerStarted","Data":"c7ea5cbd052ca66ce6d91c91b098f02ff768c030d389aa3c279c3e4887b52e05"}
Oct 30 12:02:44.394021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:44.393933    5060 generic.go:356] "Generic (PLEG): container finished" podID="9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" containerID="5c33a682dc017a5932afd5bb2f8a97fefe44a59962ef0bb9608424700ea327da" exitCode=0
Oct 30 12:02:44.394423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:44.394042    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8" event={"ID":"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69","Type":"ContainerDied","Data":"5c33a682dc017a5932afd5bb2f8a97fefe44a59962ef0bb9608424700ea327da"}
Oct 30 12:02:44.609764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:44.609725    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:45.567302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.567269    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:45.607052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.607009    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca\") pod \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") "
Oct 30 12:02:45.607268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.607083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca\") pod \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") "
Oct 30 12:02:45.607268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.607122    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx\" (UniqueName: \"kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx\") pod \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\" (UID: \"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69\") "
Oct 30 12:02:45.607507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.607443    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca" pod "9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" (UID: "9b8b12b2-6f87-4f90-b1ec-ef94c251aa69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:45.607807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.607775    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:45.609501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.609476    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca" pod "9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" (UID: "9b8b12b2-6f87-4f90-b1ec-ef94c251aa69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:02:45.609612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.609588    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx" pod "9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" (UID: "9b8b12b2-6f87-4f90-b1ec-ef94c251aa69"). InnerVolumeSpecName "kube-api-access-qzgmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:45.708810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.708773    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:45.708810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.708804    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgmx\" (UniqueName: \"kubernetes.io/projected/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69-kube-api-access-qzgmx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:45.973212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.973120    5060 patch_prober.go:29] interesting pod/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:02:45.973212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:45.973194    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:02:46.416223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:46.416074    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"
Oct 30 12:02:46.416223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:46.416101    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8" event={"ID":"9b8b12b2-6f87-4f90-b1ec-ef94c251aa69","Type":"ContainerDied","Data":"c7ea5cbd052ca66ce6d91c91b098f02ff768c030d389aa3c279c3e4887b52e05"}
Oct 30 12:02:46.416223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:46.416161    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ea5cbd052ca66ce6d91c91b098f02ff768c030d389aa3c279c3e4887b52e05"
Oct 30 12:02:46.451205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:46.451172    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:47.215582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.215553    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 12:02:47.328394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328352    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.328653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328515    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.328653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328575    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.328653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.328829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328663    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.328829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328698    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.328829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:47.328829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328744    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.329053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.328910    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.329053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.329006    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket\") pod \"d6b9dfb1-f962-4e38-8c48-989fb1478455\" (UID: \"d6b9dfb1-f962-4e38-8c48-989fb1478455\") "
Oct 30 12:02:47.329184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.329162    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:47.329279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.329260    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:47.329344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.329317    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:47.329572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.329548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:47.330054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.330028    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.330147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.330058    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.330147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.330075    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.330147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.330090    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.330147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.330104    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b9dfb1-f962-4e38-8c48-989fb1478455-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.331547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.331521    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:47.334922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.334893    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:47.334922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.334901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:02:47.335098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.334965    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6" pod "d6b9dfb1-f962-4e38-8c48-989fb1478455" (UID: "d6b9dfb1-f962-4e38-8c48-989fb1478455"). InnerVolumeSpecName "kube-api-access-76nl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:47.427364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.427325    5060 generic.go:356] "Generic (PLEG): container finished" podID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerID="a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4" exitCode=0
Oct 30 12:02:47.427364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.427373    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"
Oct 30 12:02:47.427600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.427391    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" event={"ID":"d6b9dfb1-f962-4e38-8c48-989fb1478455","Type":"ContainerDied","Data":"a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4"}
Oct 30 12:02:47.427600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.427472    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52" event={"ID":"d6b9dfb1-f962-4e38-8c48-989fb1478455","Type":"ContainerDied","Data":"259e593eea7087a4f0f79b082850b65e1903c27d16db782d649a52d5fd522c59"}
Oct 30 12:02:47.427600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.427504    5060 scope.go:122] "RemoveContainer" containerID="a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4"
Oct 30 12:02:47.431199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.431172    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.431360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.431203    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nl6\" (UniqueName: \"kubernetes.io/projected/d6b9dfb1-f962-4e38-8c48-989fb1478455-kube-api-access-76nl6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.431360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.431219    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.431360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.431233    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b9dfb1-f962-4e38-8c48-989fb1478455-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:02:47.439801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.439777    5060 scope.go:122] "RemoveContainer" containerID="e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"
Oct 30 12:02:47.450977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.450954    5060 scope.go:122] "RemoveContainer" containerID="a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4"
Oct 30 12:02:47.451354 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:02:47.451332    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4\": container with ID starting with a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4 not found: ID does not exist" containerID="a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4"
Oct 30 12:02:47.451405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.451366    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4"} err="failed to get container status \"a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4\": rpc error: code = NotFound desc = could not find container \"a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4\": container with ID starting with a5a48ab5ea91920570ec0127037153db94484afa5ee63a6353e937b1bb3193e4 not found: ID does not exist"
Oct 30 12:02:47.451405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.451386    5060 scope.go:122] "RemoveContainer" containerID="e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"
Oct 30 12:02:47.451674 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:02:47.451655    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237\": container with ID starting with e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237 not found: ID does not exist" containerID="e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"
Oct 30 12:02:47.451741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.451683    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237"} err="failed to get container status \"e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237\": rpc error: code = NotFound desc = could not find container \"e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237\": container with ID starting with e2be936eb4b355ff3143a3b4c16c85db6be9436c45edbea0dfdcf0c32a772237 not found: ID does not exist"
Oct 30 12:02:47.486803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.486717    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"]
Oct 30 12:02:47.499245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:47.499208    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-48e1f1-openshift-default-695d7746df-fxx52"]
Oct 30 12:02:48.328057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:48.328011    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:48.340941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:48.340710    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" path="/var/lib/kubelet/pods/d6b9dfb1-f962-4e38-8c48-989fb1478455/volumes"
Oct 30 12:02:49.857606 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:02:49.857567    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:02:50.218567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:50.218529    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:50.635120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:50.635021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:51.295270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:51.295217    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:52.881321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:52.881239    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:54.752582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:54.752529    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:56.660339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:56.660302    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:02:57.828408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:57.828376    5060 scope.go:122] "RemoveContainer" containerID="bc7cd897b49f3a84f5aa1fff93ce8627d3ec6c3a01b2a3f067d183e78763cd49"
Oct 30 12:02:57.839169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:57.839151    5060 scope.go:122] "RemoveContainer" containerID="a10cdca73403c8abd1a0361f82ca7dc55fde7cf542ee622d0e96486186eeeb3d"
Oct 30 12:02:58.203220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:02:58.203172    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:00.077188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:00.077136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:00.357201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:00.357101    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:02.092745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:02.092699    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:03.036446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:03.036401    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"]
Oct 30 12:03:03.045267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:03.045236    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc062-mlqcp"]
Oct 30 12:03:04.003599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:04.003557    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:04.335667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:04.335586    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7031cfb-2a81-4221-9912-2fde25a09226" path="/var/lib/kubelet/pods/a7031cfb-2a81-4221-9912-2fde25a09226/volumes"
Oct 30 12:03:05.824500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.824445    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:05.836520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.836479    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"]
Oct 30 12:03:05.837052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837035    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy"
Oct 30 12:03:05.837126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837054    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy"
Oct 30 12:03:05.837126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837070    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy"
Oct 30 12:03:05.837126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837076    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy"
Oct 30 12:03:05.837126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837092    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" containerName="revoke-keys"
Oct 30 12:03:05.837126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837097    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" containerName="revoke-keys"
Oct 30 12:03:05.837395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837185    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy"
Oct 30 12:03:05.837395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.837193    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" containerName="revoke-keys"
Oct 30 12:03:05.842759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.842730    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:05.859717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.859681    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"]
Oct 30 12:03:05.859717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.859716    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:03:05.872386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.872328    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:03:05.872386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.872337    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:03:05.961180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:05.961148    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:03:06.003029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.002988    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.003225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.003134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.003225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.003166    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vln\" (UniqueName: \"kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.104366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.104266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.104366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.104354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.104366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.104372    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vln\" (UniqueName: \"kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.105027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.105000    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.106862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.106839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.129746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.129717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vln\" (UniqueName: \"kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln\") pod \"maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.160569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.160440    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:06.477550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.477150    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"]
Oct 30 12:03:06.612373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:06.612335    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7" event={"ID":"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb","Type":"ContainerStarted","Data":"d8689bf119d2017373b59d82da993c815cb1b00cd032322184f2bdd8445e13c2"}
Oct 30 12:03:07.236199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.236168    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"]
Oct 30 12:03:07.236822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.236807    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b9dfb1-f962-4e38-8c48-989fb1478455" containerName="istio-proxy"
Oct 30 12:03:07.240417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.240393    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.292813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.292781    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"]
Oct 30 12:03:07.420705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.420675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.420866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.420722    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.420918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.420858    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.421004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.420979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfbq\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.421059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.421043    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.421188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.421167    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.421294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.421277    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.421355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.421315    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.421411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.421391    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.432986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.432933    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:07.523397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523296    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523392    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523478    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523512    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523556    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523666    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523721    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfbq\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.523982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.523792    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.524859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.524625    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.524859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.524780    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.525078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.524927    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.525396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.525353    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.527193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.527169    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.527718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.527696    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.571212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.571162    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.572916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.572856    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfbq\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.641761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.641724    5060 generic.go:356] "Generic (PLEG): container finished" podID="2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" containerID="4cf7a657eae3ea05daa2de49c3291fba19fb171f0fda7653539c689f8ed8d4e0" exitCode=0
Oct 30 12:03:07.641941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.641894    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7" event={"ID":"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb","Type":"ContainerDied","Data":"4cf7a657eae3ea05daa2de49c3291fba19fb171f0fda7653539c689f8ed8d4e0"}
Oct 30 12:03:07.659804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.659772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:07.797031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.796933    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"]
Oct 30 12:03:07.804231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.804199    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.810062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.808579    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"]
Oct 30 12:03:07.835119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.835086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.835119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.835134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnngr\" (UniqueName: \"kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.835396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.835160    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.837973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.837942    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:07.859438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.859407    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:07.939477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.937629    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.939477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.937707    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnngr\" (UniqueName: \"kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.939477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.937742    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.939477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.938637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.963477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.942106    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:07.982201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:07.982121    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnngr\" (UniqueName: \"kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr\") pod \"maas-api-revoke-sub-a5071577-bj4ct\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:08.133387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.133311    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:08.227257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.226645    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:08.287150 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:08.287114    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4861f5c_0c88_4db7_9455_1a3832272d01.slice/crio-67070b6da4a1aa404b00b818b19f9b0f40bcb998d71dea874fa4c76c0b143195 WatchSource:0}: Error finding container 67070b6da4a1aa404b00b818b19f9b0f40bcb998d71dea874fa4c76c0b143195: Status 404 returned error can't find the container with id 67070b6da4a1aa404b00b818b19f9b0f40bcb998d71dea874fa4c76c0b143195
Oct 30 12:03:08.288434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.288398    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"]
Oct 30 12:03:08.290894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.290854    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:03:08.291008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.290979    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:03:08.291081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.291026    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:03:08.357584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.357554    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"]
Oct 30 12:03:08.361396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.361370    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f11c-bwlx8"]
Oct 30 12:03:08.394845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.394758    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:08.456678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.455388    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"]
Oct 30 12:03:08.653037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.652949    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct" event={"ID":"accda864-c967-4160-b47e-a6e66ce05f92","Type":"ContainerStarted","Data":"50b75069105e1ea907cee6f7576451fd44d2414ec060173565b10b8947c1a157"}
Oct 30 12:03:08.655573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.655530    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" event={"ID":"c4861f5c-0c88-4db7-9455-1a3832272d01","Type":"ContainerStarted","Data":"831a63c2501563e21a21ff8233383971bee583cac405323d20c7d052503b867e"}
Oct 30 12:03:08.655777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.655599    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" event={"ID":"c4861f5c-0c88-4db7-9455-1a3832272d01","Type":"ContainerStarted","Data":"67070b6da4a1aa404b00b818b19f9b0f40bcb998d71dea874fa4c76c0b143195"}
Oct 30 12:03:08.710189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.710112    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podStartSLOduration=1.710090001 podStartE2EDuration="1.710090001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:08.704419395 +0000 UTC m=+1992.499303889" watchObservedRunningTime="2026-10-30 12:03:08.710090001 +0000 UTC m=+1992.504974493"
Oct 30 12:03:08.822016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.821986    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:08.860383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.860354    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:08.907703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.907612    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:08.951052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.951018    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca\") pod \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") "
Oct 30 12:03:08.951247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.951065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln\" (UniqueName: \"kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln\") pod \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") "
Oct 30 12:03:08.951247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.951103    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca\") pod \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\" (UID: \"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb\") "
Oct 30 12:03:08.951578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.951546    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca" pod "2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" (UID: "2c8c9ecb-cc55-431f-8f55-ae30cd3130bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:08.951812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.951795    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:08.953437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.953414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca" pod "2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" (UID: "2c8c9ecb-cc55-431f-8f55-ae30cd3130bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:08.953522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:08.953437    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln" pod "2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" (UID: "2c8c9ecb-cc55-431f-8f55-ae30cd3130bb"). InnerVolumeSpecName "kube-api-access-k6vln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:09.053201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.053166    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:09.053201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.053196    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vln\" (UniqueName: \"kubernetes.io/projected/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb-kube-api-access-k6vln\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:09.666339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.666308    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"
Oct 30 12:03:09.666339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.666330    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7" event={"ID":"2c8c9ecb-cc55-431f-8f55-ae30cd3130bb","Type":"ContainerDied","Data":"d8689bf119d2017373b59d82da993c815cb1b00cd032322184f2bdd8445e13c2"}
Oct 30 12:03:09.666843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.666383    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8689bf119d2017373b59d82da993c815cb1b00cd032322184f2bdd8445e13c2"
Oct 30 12:03:09.668189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.668159    5060 generic.go:356] "Generic (PLEG): container finished" podID="accda864-c967-4160-b47e-a6e66ce05f92" containerID="5898f163744810e89dc0d29d20435e4b8d5ca4b1a80c5a505645653efad6f23b" exitCode=0
Oct 30 12:03:09.668337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.668252    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct" event={"ID":"accda864-c967-4160-b47e-a6e66ce05f92","Type":"ContainerDied","Data":"5898f163744810e89dc0d29d20435e4b8d5ca4b1a80c5a505645653efad6f23b"}
Oct 30 12:03:09.862064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.861977    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:09.862064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:09.862031    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:10.343262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.343228    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8b12b2-6f87-4f90-b1ec-ef94c251aa69" path="/var/lib/kubelet/pods/9b8b12b2-6f87-4f90-b1ec-ef94c251aa69/volumes"
Oct 30 12:03:10.531685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.531645    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:10.860328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.860296    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:10.860823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.860347    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:10.916009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.915981    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:10.999484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.990916    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca\") pod \"accda864-c967-4160-b47e-a6e66ce05f92\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") "
Oct 30 12:03:10.999912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:10.999882    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr\" (UniqueName: \"kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr\") pod \"accda864-c967-4160-b47e-a6e66ce05f92\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") "
Oct 30 12:03:11.007478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.000112    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca\") pod \"accda864-c967-4160-b47e-a6e66ce05f92\" (UID: \"accda864-c967-4160-b47e-a6e66ce05f92\") "
Oct 30 12:03:11.010630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.000622    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca" pod "accda864-c967-4160-b47e-a6e66ce05f92" (UID: "accda864-c967-4160-b47e-a6e66ce05f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:11.011152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.011123    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:11.016265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.016210    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr" pod "accda864-c967-4160-b47e-a6e66ce05f92" (UID: "accda864-c967-4160-b47e-a6e66ce05f92"). InnerVolumeSpecName "kube-api-access-tnngr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:11.020478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.020376    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca" pod "accda864-c967-4160-b47e-a6e66ce05f92" (UID: "accda864-c967-4160-b47e-a6e66ce05f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:11.112411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.112365    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnngr\" (UniqueName: \"kubernetes.io/projected/accda864-c967-4160-b47e-a6e66ce05f92-kube-api-access-tnngr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:11.112411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.112406    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/accda864-c967-4160-b47e-a6e66ce05f92-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:11.710873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.710843    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"
Oct 30 12:03:11.710873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.710874    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct" event={"ID":"accda864-c967-4160-b47e-a6e66ce05f92","Type":"ContainerDied","Data":"50b75069105e1ea907cee6f7576451fd44d2414ec060173565b10b8947c1a157"}
Oct 30 12:03:11.711172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.710933    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b75069105e1ea907cee6f7576451fd44d2414ec060173565b10b8947c1a157"
Oct 30 12:03:11.844289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844249    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"]
Oct 30 12:03:11.844833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844815    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" containerName="revoke-keys"
Oct 30 12:03:11.844833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844830    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" containerName="revoke-keys"
Oct 30 12:03:11.844992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844862    5060 cpu_manager.go:414] "containerMap: removing container" podUID="accda864-c967-4160-b47e-a6e66ce05f92" containerName="revoke-keys"
Oct 30 12:03:11.844992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844867    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accda864-c967-4160-b47e-a6e66ce05f92" containerName="revoke-keys"
Oct 30 12:03:11.844992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844960    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" containerName="revoke-keys"
Oct 30 12:03:11.844992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.844974    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="accda864-c967-4160-b47e-a6e66ce05f92" containerName="revoke-keys"
Oct 30 12:03:11.848579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.848518    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:11.860762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.860709    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:11.861215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.860790    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:11.866541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.866499    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"]
Oct 30 12:03:11.866805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.866769    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:03:11.866805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.866769    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:03:11.866805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.866798    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:03:11.874500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:11.874467    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:03:12.024675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.024572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.024675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.024649    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.024924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.024777    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5w9f\" (UniqueName: \"kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.126212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.126162    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.126494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.126254    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.126494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.126329    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5w9f\" (UniqueName: \"kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.127123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.127093    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.129083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.129058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.151051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.151013    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5w9f\" (UniqueName: \"kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f\") pod \"maas-api-revoke-sub-b348edd2-ns9wg\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.163018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.162986    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:12.525633 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:12.525586    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7714013a_ccf5_4ec9_a7db_bfcf38a7bf6f.slice/crio-487f303870f7688e4b8ee694965f51977b4e118f2b7faa4b84f430371d4275dd WatchSource:0}: Error finding container 487f303870f7688e4b8ee694965f51977b4e118f2b7faa4b84f430371d4275dd: Status 404 returned error can't find the container with id 487f303870f7688e4b8ee694965f51977b4e118f2b7faa4b84f430371d4275dd
Oct 30 12:03:12.527615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.527582    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"]
Oct 30 12:03:12.606053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.606018    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:12.723236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.723199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg" event={"ID":"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f","Type":"ContainerStarted","Data":"487f303870f7688e4b8ee694965f51977b4e118f2b7faa4b84f430371d4275dd"}
Oct 30 12:03:12.859988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.859944    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:12.860144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:12.860019    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:13.735424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:13.735382    5060 generic.go:356] "Generic (PLEG): container finished" podID="7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" containerID="7fac8153452e850c46ff01f1ec31fedd3cb20ae760f5b244f3c3a744a14f71a9" exitCode=0
Oct 30 12:03:13.735865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:13.735479    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg" event={"ID":"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f","Type":"ContainerDied","Data":"7fac8153452e850c46ff01f1ec31fedd3cb20ae760f5b244f3c3a744a14f71a9"}
Oct 30 12:03:13.859853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:13.859820    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:13.860050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:13.859881    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:14.629000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:14.628965    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:14.860365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:14.860215    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:14.860365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:14.860268    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:14.955536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:14.955322    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:15.094213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.094178    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:15.125196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.125166    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:15.179389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.179359    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca\") pod \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") "
Oct 30 12:03:15.179606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.179399    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f\" (UniqueName: \"kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f\") pod \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") "
Oct 30 12:03:15.179680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.179592    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca\") pod \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\" (UID: \"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f\") "
Oct 30 12:03:15.179814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.179792    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca" pod "7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" (UID: "7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:15.180185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.180114    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:15.181771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.181743    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca" pod "7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" (UID: "7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:15.181961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.181941    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f" pod "7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" (UID: "7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f"). InnerVolumeSpecName "kube-api-access-z5w9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:15.277178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.264873    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"]
Oct 30 12:03:15.277178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.265128    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" containerID="cri-o://0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b" gracePeriod=30
Oct 30 12:03:15.281533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.281443    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5w9f\" (UniqueName: \"kubernetes.io/projected/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-kube-api-access-z5w9f\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:15.281533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.281490    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:15.318981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.318934    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:15.784465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.784420    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"
Oct 30 12:03:15.784676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.784441    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg" event={"ID":"7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f","Type":"ContainerDied","Data":"487f303870f7688e4b8ee694965f51977b4e118f2b7faa4b84f430371d4275dd"}
Oct 30 12:03:15.784676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.784565    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487f303870f7688e4b8ee694965f51977b4e118f2b7faa4b84f430371d4275dd"
Oct 30 12:03:15.860858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.860816    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 12:03:15.861263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.860886    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 12:03:15.940003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.939966    5060 patch_prober.go:29] interesting pod/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:03:15.940177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:15.940028    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:03:16.376856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:16.376818    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"]
Oct 30 12:03:16.388718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:16.388683    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"]
Oct 30 12:03:16.519114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:16.519069    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:16.670529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:16.670410    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:17.844355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:17.844314    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:17.845304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:17.845279    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:17.847862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:17.847839    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:03:18.449490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:18.449410    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:19.857724 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:03:19.857677    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:03:20.196199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.196157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:20.572985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.572959    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:03:20.676446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676403    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676513    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676630    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676721    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676756    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676853    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676893    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:20.676982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676904    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.676982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.676950    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data\") pod \"21d6c388-395a-494d-bbb2-f5adf053c0de\" (UID: \"21d6c388-395a-494d-bbb2-f5adf053c0de\") "
Oct 30 12:03:20.677194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.677147    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:20.677838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.677582    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:20.677838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.677606    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d6c388-395a-494d-bbb2-f5adf053c0de-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.677838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.677628    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.678099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.677849    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:20.678573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.678551    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:20.680409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.680380    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:03:20.680628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.680515    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:20.681219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.681193    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:20.681487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.681424    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7" pod "21d6c388-395a-494d-bbb2-f5adf053c0de" (UID: "21d6c388-395a-494d-bbb2-f5adf053c0de"). InnerVolumeSpecName "kube-api-access-48rm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:20.778811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778714    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rm7\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-kube-api-access-48rm7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.778811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778757    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.778811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778772    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.778811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778788    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.778811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778803    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d6c388-395a-494d-bbb2-f5adf053c0de-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.778811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778817    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.779124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.778830    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d6c388-395a-494d-bbb2-f5adf053c0de-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:20.832051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.832018    5060 generic.go:356] "Generic (PLEG): container finished" podID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerID="0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b" exitCode=0
Oct 30 12:03:20.832242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.832071    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"
Oct 30 12:03:20.832242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.832102    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" event={"ID":"21d6c388-395a-494d-bbb2-f5adf053c0de","Type":"ContainerDied","Data":"0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b"}
Oct 30 12:03:20.832242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.832163    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv" event={"ID":"21d6c388-395a-494d-bbb2-f5adf053c0de","Type":"ContainerDied","Data":"95d1800501b09c06f33fe4d3675bcd249ce038b591ee79ad6e9b305855d337c1"}
Oct 30 12:03:20.832242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.832193    5060 scope.go:122] "RemoveContainer" containerID="0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b"
Oct 30 12:03:20.844492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.844472    5060 scope.go:122] "RemoveContainer" containerID="2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"
Oct 30 12:03:20.856135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.856108    5060 scope.go:122] "RemoveContainer" containerID="0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b"
Oct 30 12:03:20.856479 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:03:20.856445    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b\": container with ID starting with 0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b not found: ID does not exist" containerID="0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b"
Oct 30 12:03:20.856541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.856497    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b"} err="failed to get container status \"0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b\": rpc error: code = NotFound desc = could not find container \"0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b\": container with ID starting with 0fc849742a12faa8e2858230cb568390be54e0bd19788b7ec027adbfd19cf42b not found: ID does not exist"
Oct 30 12:03:20.856541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.856514    5060 scope.go:122] "RemoveContainer" containerID="2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"
Oct 30 12:03:20.856820 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:03:20.856799    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72\": container with ID starting with 2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72 not found: ID does not exist" containerID="2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"
Oct 30 12:03:20.856871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.856826    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72"} err="failed to get container status \"2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72\": rpc error: code = NotFound desc = could not find container \"2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72\": container with ID starting with 2fb640fcedce505190156699783701f3179db89c03c1dc41d5d2406039aabd72 not found: ID does not exist"
Oct 30 12:03:20.888856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.888821    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"]
Oct 30 12:03:20.892642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:20.892613    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f11c-openshift-default-749c888765-rtgjv"]
Oct 30 12:03:21.377115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:21.377067    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerName="payload-processing" containerID="cri-o://34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d" gracePeriod=25
Oct 30 12:03:21.389881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:21.389837    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerName="payload-pre-processing" containerID="cri-o://0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c" gracePeriod=25
Oct 30 12:03:21.944206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:21.944170    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:22.335818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:22.335780    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" path="/var/lib/kubelet/pods/21d6c388-395a-494d-bbb2-f5adf053c0de/volumes"
Oct 30 12:03:23.760417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:23.760379    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:25.206301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206252    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"]
Oct 30 12:03:25.206885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206795    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" containerName="revoke-keys"
Oct 30 12:03:25.206885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206809    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" containerName="revoke-keys"
Oct 30 12:03:25.206885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206841    5060 cpu_manager.go:414] "containerMap: removing container" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy"
Oct 30 12:03:25.206885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206846    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy"
Oct 30 12:03:25.206885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206858    5060 cpu_manager.go:414] "containerMap: removing container" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy"
Oct 30 12:03:25.206885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206863    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy"
Oct 30 12:03:25.207307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206946    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy"
Oct 30 12:03:25.207307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206957    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d6c388-395a-494d-bbb2-f5adf053c0de" containerName="istio-proxy"
Oct 30 12:03:25.207307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.206965    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" containerName="revoke-keys"
Oct 30 12:03:25.211551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.211525    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.231191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.231148    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"]
Oct 30 12:03:25.237782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.237745    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:03:25.237998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.237803    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:03:25.237998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.237884    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:03:25.238892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.238869    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:03:25.332698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.332658    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.332918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.332794    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.332991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.332935    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgp75\" (UniqueName: \"kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.434208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.434170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.434413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.434258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgp75\" (UniqueName: \"kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.434413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.434325    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.434935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.434913    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.436896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.436874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.464185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.464107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgp75\" (UniqueName: \"kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75\") pod \"maas-api-revoke-sub-5a7ca321-5c2lr\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.527032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.526994    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:25.578305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.578265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:25.793529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.793416    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:03:25.793529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.793523    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:03:25.839495 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:25.839445    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2265fec5_69bb_4b0f_b581_01b8b9d53903.slice/crio-2ef1b55710feaa48368406a3ba4eb6f54fef9793b487885916fdbf6db971ceb1 WatchSource:0}: Error finding container 2ef1b55710feaa48368406a3ba4eb6f54fef9793b487885916fdbf6db971ceb1: Status 404 returned error can't find the container with id 2ef1b55710feaa48368406a3ba4eb6f54fef9793b487885916fdbf6db971ceb1
Oct 30 12:03:25.841662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.841634    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"]
Oct 30 12:03:25.872088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:25.872054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr" event={"ID":"2265fec5-69bb-4b0f-b581-01b8b9d53903","Type":"ContainerStarted","Data":"2ef1b55710feaa48368406a3ba4eb6f54fef9793b487885916fdbf6db971ceb1"}
Oct 30 12:03:26.883663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:26.883627    5060 generic.go:356] "Generic (PLEG): container finished" podID="2265fec5-69bb-4b0f-b581-01b8b9d53903" containerID="c90b5e27fd91962d8b2a9d2754fbb437cd3e0bff435a29ba3f7fa7acdbfcf53d" exitCode=0
Oct 30 12:03:26.884048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:26.883711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr" event={"ID":"2265fec5-69bb-4b0f-b581-01b8b9d53903","Type":"ContainerDied","Data":"c90b5e27fd91962d8b2a9d2754fbb437cd3e0bff435a29ba3f7fa7acdbfcf53d"}
Oct 30 12:03:27.396198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:27.396156    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:28.734569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.662683    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:28.734569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.730588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca\") pod \"2265fec5-69bb-4b0f-b581-01b8b9d53903\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") "
Oct 30 12:03:28.734569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.730670    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75\" (UniqueName: \"kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75\") pod \"2265fec5-69bb-4b0f-b581-01b8b9d53903\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") "
Oct 30 12:03:28.734569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.730719    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca\") pod \"2265fec5-69bb-4b0f-b581-01b8b9d53903\" (UID: \"2265fec5-69bb-4b0f-b581-01b8b9d53903\") "
Oct 30 12:03:28.734569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.732850    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca" pod "2265fec5-69bb-4b0f-b581-01b8b9d53903" (UID: "2265fec5-69bb-4b0f-b581-01b8b9d53903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:28.765523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.763935    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75" pod "2265fec5-69bb-4b0f-b581-01b8b9d53903" (UID: "2265fec5-69bb-4b0f-b581-01b8b9d53903"). InnerVolumeSpecName "kube-api-access-qgp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:28.765523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.764043    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca" pod "2265fec5-69bb-4b0f-b581-01b8b9d53903" (UID: "2265fec5-69bb-4b0f-b581-01b8b9d53903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:28.832516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.832445    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:28.832516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.832511    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgp75\" (UniqueName: \"kubernetes.io/projected/2265fec5-69bb-4b0f-b581-01b8b9d53903-kube-api-access-qgp75\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:28.832516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.832522    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265fec5-69bb-4b0f-b581-01b8b9d53903-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:28.918689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.918656    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"
Oct 30 12:03:28.918883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.918687    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr" event={"ID":"2265fec5-69bb-4b0f-b581-01b8b9d53903","Type":"ContainerDied","Data":"2ef1b55710feaa48368406a3ba4eb6f54fef9793b487885916fdbf6db971ceb1"}
Oct 30 12:03:28.918883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:28.918748    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ef1b55710feaa48368406a3ba4eb6f54fef9793b487885916fdbf6db971ceb1"
Oct 30 12:03:29.482231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:29.482179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:31.260530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:31.260479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:33.058789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:33.058747    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:34.841882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:34.841833    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:35.038816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.038777    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"]
Oct 30 12:03:35.039335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.039322    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2265fec5-69bb-4b0f-b581-01b8b9d53903" containerName="revoke-keys"
Oct 30 12:03:35.039382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.039338    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2265fec5-69bb-4b0f-b581-01b8b9d53903" containerName="revoke-keys"
Oct 30 12:03:35.039446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.039437    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2265fec5-69bb-4b0f-b581-01b8b9d53903" containerName="revoke-keys"
Oct 30 12:03:35.043161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.043140    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.061234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.061205    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"]
Oct 30 12:03:35.072736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.068253    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:03:35.072736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.068278    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:03:35.072736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.068376    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:03:35.072736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.069017    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:03:35.111068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.110979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.111068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.111024    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br68s\" (UniqueName: \"kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.111288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.111146    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.212262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.212222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.212262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.212266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br68s\" (UniqueName: \"kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.212628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.212297    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.213032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.213002    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.214972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.214947    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.236666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.236630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br68s\" (UniqueName: \"kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s\") pod \"maas-api-revoke-sub-271932b4-2rczb\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.357901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.357865    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:35.680405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.680371    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"]
Oct 30 12:03:35.688219 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:35.688179    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc8e275_321c_4ce9_bf96_88481100310b.slice/crio-566345bbdc7a12d7f45656fc66b5c67fcfed0b3676b4f4634dcd1cc058ef9c11 WatchSource:0}: Error finding container 566345bbdc7a12d7f45656fc66b5c67fcfed0b3676b4f4634dcd1cc058ef9c11: Status 404 returned error can't find the container with id 566345bbdc7a12d7f45656fc66b5c67fcfed0b3676b4f4634dcd1cc058ef9c11
Oct 30 12:03:35.793403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.793354    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:03:35.793629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.793445    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:03:35.983062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:35.983023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb" event={"ID":"3fc8e275-321c-4ce9-bf96-88481100310b","Type":"ContainerStarted","Data":"566345bbdc7a12d7f45656fc66b5c67fcfed0b3676b4f4634dcd1cc058ef9c11"}
Oct 30 12:03:36.307430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.307296    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"]
Oct 30 12:03:36.315699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.315648    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.340661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.340624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"]
Oct 30 12:03:36.429498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.429464    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnjs\" (UniqueName: \"kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.429726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.429507    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.429798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.429758    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.531272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.531227    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.531439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.531313    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnjs\" (UniqueName: \"kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.531439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.531336    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.532044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.532016    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.533988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.533964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.556412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.556383    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnjs\" (UniqueName: \"kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs\") pod \"maas-api-revoke-sub-af089860-p6m6j\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.638374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.638274    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:36.675885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.675849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:36.979925 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:36.979889    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49f799c_806a_48d0_b3b7_5485810bb37f.slice/crio-89702502d32a7b9de490933d20f2ef09a93cf886a8ec4906ec89b0b7c9337802 WatchSource:0}: Error finding container 89702502d32a7b9de490933d20f2ef09a93cf886a8ec4906ec89b0b7c9337802: Status 404 returned error can't find the container with id 89702502d32a7b9de490933d20f2ef09a93cf886a8ec4906ec89b0b7c9337802
Oct 30 12:03:36.981489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.981446    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"]
Oct 30 12:03:36.981655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.981526    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:36.995928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.995887    5060 generic.go:356] "Generic (PLEG): container finished" podID="3fc8e275-321c-4ce9-bf96-88481100310b" containerID="24d983a3840d42591cc48136f202d563e34ddad4b0258109a85d6ecc9f8ca535" exitCode=0
Oct 30 12:03:36.996418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.995979    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb" event={"ID":"3fc8e275-321c-4ce9-bf96-88481100310b","Type":"ContainerDied","Data":"24d983a3840d42591cc48136f202d563e34ddad4b0258109a85d6ecc9f8ca535"}
Oct 30 12:03:36.997876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:36.997847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j" event={"ID":"d49f799c-806a-48d0-b3b7-5485810bb37f","Type":"ContainerStarted","Data":"89702502d32a7b9de490933d20f2ef09a93cf886a8ec4906ec89b0b7c9337802"}
Oct 30 12:03:37.476905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:37.476867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:37.780439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:37.780398    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:38.013685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.013645    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j" event={"ID":"d49f799c-806a-48d0-b3b7-5485810bb37f","Type":"ContainerStarted","Data":"f1a565946910b96bfc534b62b310615c7e4ea144900240361a43029172ab7eae"}
Oct 30 12:03:38.180788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.180756    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:38.259287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.259244    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s\" (UniqueName: \"kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s\") pod \"3fc8e275-321c-4ce9-bf96-88481100310b\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") "
Oct 30 12:03:38.259515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.259316    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca\") pod \"3fc8e275-321c-4ce9-bf96-88481100310b\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") "
Oct 30 12:03:38.259515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.259361    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca\") pod \"3fc8e275-321c-4ce9-bf96-88481100310b\" (UID: \"3fc8e275-321c-4ce9-bf96-88481100310b\") "
Oct 30 12:03:38.259909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.259867    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca" pod "3fc8e275-321c-4ce9-bf96-88481100310b" (UID: "3fc8e275-321c-4ce9-bf96-88481100310b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:38.260119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.260093    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:38.262598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.262570    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca" pod "3fc8e275-321c-4ce9-bf96-88481100310b" (UID: "3fc8e275-321c-4ce9-bf96-88481100310b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:38.262798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.262768    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s" pod "3fc8e275-321c-4ce9-bf96-88481100310b" (UID: "3fc8e275-321c-4ce9-bf96-88481100310b"). InnerVolumeSpecName "kube-api-access-br68s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:38.361981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.361938    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br68s\" (UniqueName: \"kubernetes.io/projected/3fc8e275-321c-4ce9-bf96-88481100310b-kube-api-access-br68s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:38.361981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.361976    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8e275-321c-4ce9-bf96-88481100310b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:38.555568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:38.555521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:39.023711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.023676    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"
Oct 30 12:03:39.024237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.023715    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb" event={"ID":"3fc8e275-321c-4ce9-bf96-88481100310b","Type":"ContainerDied","Data":"566345bbdc7a12d7f45656fc66b5c67fcfed0b3676b4f4634dcd1cc058ef9c11"}
Oct 30 12:03:39.024237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.023777    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="566345bbdc7a12d7f45656fc66b5c67fcfed0b3676b4f4634dcd1cc058ef9c11"
Oct 30 12:03:39.025799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.025770    5060 generic.go:356] "Generic (PLEG): container finished" podID="d49f799c-806a-48d0-b3b7-5485810bb37f" containerID="f1a565946910b96bfc534b62b310615c7e4ea144900240361a43029172ab7eae" exitCode=0
Oct 30 12:03:39.025968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.025864    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j" event={"ID":"d49f799c-806a-48d0-b3b7-5485810bb37f","Type":"ContainerDied","Data":"f1a565946910b96bfc534b62b310615c7e4ea144900240361a43029172ab7eae"}
Oct 30 12:03:39.216006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.215965    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:39.276326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.276233    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca\") pod \"d49f799c-806a-48d0-b3b7-5485810bb37f\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") "
Oct 30 12:03:39.276791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.276749    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca" pod "d49f799c-806a-48d0-b3b7-5485810bb37f" (UID: "d49f799c-806a-48d0-b3b7-5485810bb37f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:39.276791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.276755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs\" (UniqueName: \"kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs\") pod \"d49f799c-806a-48d0-b3b7-5485810bb37f\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") "
Oct 30 12:03:39.276987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.276954    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca\") pod \"d49f799c-806a-48d0-b3b7-5485810bb37f\" (UID: \"d49f799c-806a-48d0-b3b7-5485810bb37f\") "
Oct 30 12:03:39.277578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.277552    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:39.280159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.280095    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs" pod "d49f799c-806a-48d0-b3b7-5485810bb37f" (UID: "d49f799c-806a-48d0-b3b7-5485810bb37f"). InnerVolumeSpecName "kube-api-access-znnjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:39.282033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.282004    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca" pod "d49f799c-806a-48d0-b3b7-5485810bb37f" (UID: "d49f799c-806a-48d0-b3b7-5485810bb37f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:39.378662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.378614    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnjs\" (UniqueName: \"kubernetes.io/projected/d49f799c-806a-48d0-b3b7-5485810bb37f-kube-api-access-znnjs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:39.378662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:39.378643    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49f799c-806a-48d0-b3b7-5485810bb37f-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:40.036368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:40.036341    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"
Oct 30 12:03:40.036834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:40.036375    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j" event={"ID":"d49f799c-806a-48d0-b3b7-5485810bb37f","Type":"ContainerDied","Data":"89702502d32a7b9de490933d20f2ef09a93cf886a8ec4906ec89b0b7c9337802"}
Oct 30 12:03:40.036834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:40.036419    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89702502d32a7b9de490933d20f2ef09a93cf886a8ec4906ec89b0b7c9337802"
Oct 30 12:03:40.360942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:40.360842    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:41.583967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.583921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:41.763164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.763134    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 12:03:41.766412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.766387    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.825894    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume\") pod \"4211a8d7-2b8a-48d4-b747-09beb30621b0\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") "
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.826015    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85\" (UniqueName: \"kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85\") pod \"4211a8d7-2b8a-48d4-b747-09beb30621b0\" (UID: \"4211a8d7-2b8a-48d4-b747-09beb30621b0\") "
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.826117    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume\") pod \"e55df9ef-cccb-41c5-a82f-841bd60973dd\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") "
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.826157    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9\" (UniqueName: \"kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9\") pod \"e55df9ef-cccb-41c5-a82f-841bd60973dd\" (UID: \"e55df9ef-cccb-41c5-a82f-841bd60973dd\") "
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.830131    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume" pod "4211a8d7-2b8a-48d4-b747-09beb30621b0" (UID: "4211a8d7-2b8a-48d4-b747-09beb30621b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.830305    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume" pod "e55df9ef-cccb-41c5-a82f-841bd60973dd" (UID: "e55df9ef-cccb-41c5-a82f-841bd60973dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.832406    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85" pod "4211a8d7-2b8a-48d4-b747-09beb30621b0" (UID: "4211a8d7-2b8a-48d4-b747-09beb30621b0"). InnerVolumeSpecName "kube-api-access-55q85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:41.835521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.833896    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9" pod "e55df9ef-cccb-41c5-a82f-841bd60973dd" (UID: "e55df9ef-cccb-41c5-a82f-841bd60973dd"). InnerVolumeSpecName "kube-api-access-tdjm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:41.861572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.861525    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:41.949079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.946445    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55df9ef-cccb-41c5-a82f-841bd60973dd-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:41.949079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.946528    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdjm9\" (UniqueName: \"kubernetes.io/projected/e55df9ef-cccb-41c5-a82f-841bd60973dd-kube-api-access-tdjm9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:41.949079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.946543    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4211a8d7-2b8a-48d4-b747-09beb30621b0-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:41.949079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:41.946560    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55q85\" (UniqueName: \"kubernetes.io/projected/4211a8d7-2b8a-48d4-b747-09beb30621b0-kube-api-access-55q85\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:42.057616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.057575    5060 generic.go:356] "Generic (PLEG): container finished" podID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerID="34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d" exitCode=0
Oct 30 12:03:42.057804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.057752    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" event={"ID":"4211a8d7-2b8a-48d4-b747-09beb30621b0","Type":"ContainerDied","Data":"34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d"}
Oct 30 12:03:42.057804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.057793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs" event={"ID":"4211a8d7-2b8a-48d4-b747-09beb30621b0","Type":"ContainerDied","Data":"39338a39018d70e22ccc33df9c0cdb6e81a3c651d6fb44a45de8547ddf83ab39"}
Oct 30 12:03:42.057882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.057814    5060 scope.go:122] "RemoveContainer" containerID="34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d"
Oct 30 12:03:42.057948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.057936    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"
Oct 30 12:03:42.061214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.061181    5060 generic.go:356] "Generic (PLEG): container finished" podID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerID="0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c" exitCode=0
Oct 30 12:03:42.061338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.061218    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" event={"ID":"e55df9ef-cccb-41c5-a82f-841bd60973dd","Type":"ContainerDied","Data":"0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c"}
Oct 30 12:03:42.061338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.061256    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"
Oct 30 12:03:42.061338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.061281    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q" event={"ID":"e55df9ef-cccb-41c5-a82f-841bd60973dd","Type":"ContainerDied","Data":"487de1283ddf79737fe77a70b712f48a3f0e9bbc87d4cc3a157506b0cb862dcc"}
Oct 30 12:03:42.071524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.071506    5060 scope.go:122] "RemoveContainer" containerID="34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d"
Oct 30 12:03:42.071902 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:03:42.071871    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d\": container with ID starting with 34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d not found: ID does not exist" containerID="34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d"
Oct 30 12:03:42.071988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.071912    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d"} err="failed to get container status \"34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d\": rpc error: code = NotFound desc = could not find container \"34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d\": container with ID starting with 34bf5f350a13386f0c18036bf879de7b30f77f0eeedafd5942105dad226e063d not found: ID does not exist"
Oct 30 12:03:42.071988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.071932    5060 scope.go:122] "RemoveContainer" containerID="0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c"
Oct 30 12:03:42.127995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.127136    5060 scope.go:122] "RemoveContainer" containerID="0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c"
Oct 30 12:03:42.127995 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:03:42.127630    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c\": container with ID starting with 0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c not found: ID does not exist" containerID="0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c"
Oct 30 12:03:42.127995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.127668    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c"} err="failed to get container status \"0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c\": rpc error: code = NotFound desc = could not find container \"0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c\": container with ID starting with 0d42c1f64e9d5b184f7d4f05b637981f02f83d55e3b435707138ee8b506be56c not found: ID does not exist"
Oct 30 12:03:42.154015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.153937    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"]
Oct 30 12:03:42.161867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.161829    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e25f4a-5575b57bb7-zft7q"]
Oct 30 12:03:42.192505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.189713    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"]
Oct 30 12:03:42.200536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.196923    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e25f4a-6dc7b89886-cqwcs"]
Oct 30 12:03:42.353483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.353420    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" path="/var/lib/kubelet/pods/4211a8d7-2b8a-48d4-b747-09beb30621b0/volumes"
Oct 30 12:03:42.354334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.354311    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" path="/var/lib/kubelet/pods/e55df9ef-cccb-41c5-a82f-841bd60973dd/volumes"
Oct 30 12:03:42.966504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.966471    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"]
Oct 30 12:03:42.967001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.966988    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3fc8e275-321c-4ce9-bf96-88481100310b" containerName="revoke-keys"
Oct 30 12:03:42.967048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967002    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc8e275-321c-4ce9-bf96-88481100310b" containerName="revoke-keys"
Oct 30 12:03:42.967048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967044    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d49f799c-806a-48d0-b3b7-5485810bb37f" containerName="revoke-keys"
Oct 30 12:03:42.967116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967050    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49f799c-806a-48d0-b3b7-5485810bb37f" containerName="revoke-keys"
Oct 30 12:03:42.967116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967057    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerName="payload-processing"
Oct 30 12:03:42.967116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967062    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerName="payload-processing"
Oct 30 12:03:42.967116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967071    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerName="payload-pre-processing"
Oct 30 12:03:42.967116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967077    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerName="payload-pre-processing"
Oct 30 12:03:42.967274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967151    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc8e275-321c-4ce9-bf96-88481100310b" containerName="revoke-keys"
Oct 30 12:03:42.967274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967166    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49f799c-806a-48d0-b3b7-5485810bb37f" containerName="revoke-keys"
Oct 30 12:03:42.967274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967173    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4211a8d7-2b8a-48d4-b747-09beb30621b0" containerName="payload-processing"
Oct 30 12:03:42.967274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.967179    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55df9ef-cccb-41c5-a82f-841bd60973dd" containerName="payload-pre-processing"
Oct 30 12:03:42.971965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.971938    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:42.992799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.992762    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"]
Oct 30 12:03:42.998934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.998888    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:03:42.999257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.999061    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:03:42.999257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:42.999172    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:03:43.001710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.001678    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:43.011310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.011274    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:03:43.062435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.062399    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.062435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.062437    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxzg\" (UniqueName: \"kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.062937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.062581    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.163477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.163429    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.163672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.163514    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxzg\" (UniqueName: \"kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.163672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.163570    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.164236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.164216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.166057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.166035    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.190845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.190811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxzg\" (UniqueName: \"kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg\") pod \"maas-api-revoke-sub-6ce9d129-pq2wc\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.286314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.286217    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:43.618262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.618220    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"]
Oct 30 12:03:43.629410 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:43.629350    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fdce63_1b85_42fd_a08e_765ff9a33029.slice/crio-6a2752943665571878c29aa65b97a9f64443a5cd1bdf8a4c1ad35a017ad6756f WatchSource:0}: Error finding container 6a2752943665571878c29aa65b97a9f64443a5cd1bdf8a4c1ad35a017ad6756f: Status 404 returned error can't find the container with id 6a2752943665571878c29aa65b97a9f64443a5cd1bdf8a4c1ad35a017ad6756f
Oct 30 12:03:43.737725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.737693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:43.917787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:43.917687    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:44.088559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:44.088506    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc" event={"ID":"31fdce63-1b85-42fd-a08e-765ff9a33029","Type":"ContainerStarted","Data":"6a2752943665571878c29aa65b97a9f64443a5cd1bdf8a4c1ad35a017ad6756f"}
Oct 30 12:03:44.849226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:44.849183    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:45.118319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.118225    5060 generic.go:356] "Generic (PLEG): container finished" podID="31fdce63-1b85-42fd-a08e-765ff9a33029" containerID="a9fdf14829eefe2f81a703909b9c07513b77adf279003dd426fec480dfb469d9" exitCode=0
Oct 30 12:03:45.118854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.118399    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc" event={"ID":"31fdce63-1b85-42fd-a08e-765ff9a33029","Type":"ContainerDied","Data":"a9fdf14829eefe2f81a703909b9c07513b77adf279003dd426fec480dfb469d9"}
Oct 30 12:03:45.461874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.461800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:45.628852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.628794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:45.635262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.635225    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"]
Oct 30 12:03:45.641304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.641271    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e25f4a-tv8z7"]
Oct 30 12:03:45.799523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:45.799425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:46.188906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.188861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:46.302954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.302925    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:46.335992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.335955    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8c9ecb-cc55-431f-8f55-ae30cd3130bb" path="/var/lib/kubelet/pods/2c8c9ecb-cc55-431f-8f55-ae30cd3130bb/volumes"
Oct 30 12:03:46.432489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.432358    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg\" (UniqueName: \"kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg\") pod \"31fdce63-1b85-42fd-a08e-765ff9a33029\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") "
Oct 30 12:03:46.432677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.432612    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca\") pod \"31fdce63-1b85-42fd-a08e-765ff9a33029\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") "
Oct 30 12:03:46.432677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.432663    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca\") pod \"31fdce63-1b85-42fd-a08e-765ff9a33029\" (UID: \"31fdce63-1b85-42fd-a08e-765ff9a33029\") "
Oct 30 12:03:46.433040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.433008    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca" pod "31fdce63-1b85-42fd-a08e-765ff9a33029" (UID: "31fdce63-1b85-42fd-a08e-765ff9a33029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:46.434798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.434769    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg" pod "31fdce63-1b85-42fd-a08e-765ff9a33029" (UID: "31fdce63-1b85-42fd-a08e-765ff9a33029"). InnerVolumeSpecName "kube-api-access-fjxzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:46.435060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.435039    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca" pod "31fdce63-1b85-42fd-a08e-765ff9a33029" (UID: "31fdce63-1b85-42fd-a08e-765ff9a33029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:46.534223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.534183    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:46.534223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.534218    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxzg\" (UniqueName: \"kubernetes.io/projected/31fdce63-1b85-42fd-a08e-765ff9a33029-kube-api-access-fjxzg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:46.534223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:46.534228    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fdce63-1b85-42fd-a08e-765ff9a33029-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:47.137852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:47.137818    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"
Oct 30 12:03:47.137852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:47.137847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc" event={"ID":"31fdce63-1b85-42fd-a08e-765ff9a33029","Type":"ContainerDied","Data":"6a2752943665571878c29aa65b97a9f64443a5cd1bdf8a4c1ad35a017ad6756f"}
Oct 30 12:03:47.138094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:47.137896    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a2752943665571878c29aa65b97a9f64443a5cd1bdf8a4c1ad35a017ad6756f"
Oct 30 12:03:47.777380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:47.777343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:49.531035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:49.530979    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:49.858579 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:03:49.858489    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:03:50.775555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:50.775516    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:51.103391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:51.103292    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"]
Oct 30 12:03:51.222201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:51.222153    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:51.451976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:51.451930    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:52.376029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.375992    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"]
Oct 30 12:03:52.376659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.376618    5060 cpu_manager.go:414] "containerMap: removing container" podUID="31fdce63-1b85-42fd-a08e-765ff9a33029" containerName="revoke-keys"
Oct 30 12:03:52.376659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.376633    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fdce63-1b85-42fd-a08e-765ff9a33029" containerName="revoke-keys"
Oct 30 12:03:52.376751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.376740    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fdce63-1b85-42fd-a08e-765ff9a33029" containerName="revoke-keys"
Oct 30 12:03:52.380346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.380323    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.408557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.408519    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"]
Oct 30 12:03:52.413647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.413608    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:03:52.413816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.413661    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:03:52.417556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.417530    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:03:52.417727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.417625    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:03:52.513608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.513552    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.513784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.513619    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxs5r\" (UniqueName: \"kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.513784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.513744    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.615571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.615526    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.615790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.615579    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxs5r\" (UniqueName: \"kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.615790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.615646    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.616278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.616253    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.618568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.618543    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.639783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.639708    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxs5r\" (UniqueName: \"kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r\") pod \"maas-api-revoke-sub-0458b6c7-hp2g7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:52.698221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:52.698184    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:53.012749 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:03:53.012714    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f767b3a_1fc2_4633_bdfc_feccb7e32cd7.slice/crio-b046b328e2aa0f30ba561cf674690a7ef4f65986b484c98896b495545bf63421 WatchSource:0}: Error finding container b046b328e2aa0f30ba561cf674690a7ef4f65986b484c98896b495545bf63421: Status 404 returned error can't find the container with id b046b328e2aa0f30ba561cf674690a7ef4f65986b484c98896b495545bf63421
Oct 30 12:03:53.014650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:53.014624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"]
Oct 30 12:03:53.192838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:53.192800    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7" event={"ID":"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7","Type":"ContainerStarted","Data":"b046b328e2aa0f30ba561cf674690a7ef4f65986b484c98896b495545bf63421"}
Oct 30 12:03:53.356949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:53.356901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:54.203838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.203801    5060 generic.go:356] "Generic (PLEG): container finished" podID="2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" containerID="112260326f49e509ebe71998345b828a9b706bab1719d9b65e07b1316f57e31f" exitCode=0
Oct 30 12:03:54.204259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.203883    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7" event={"ID":"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7","Type":"ContainerDied","Data":"112260326f49e509ebe71998345b828a9b706bab1719d9b65e07b1316f57e31f"}
Oct 30 12:03:54.735068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.735028    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"]
Oct 30 12:03:54.739357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.739334    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.765416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.765385    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"]
Oct 30 12:03:54.843554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.843521    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drs9l\" (UniqueName: \"kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.843874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.843683    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.843874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.843739    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.944903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.944861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.945149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.944923    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.945149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.945040    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drs9l\" (UniqueName: \"kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.945663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.945640    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.947813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.947790    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:54.969236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:54.969203    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drs9l\" (UniqueName: \"kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l\") pod \"maas-api-revoke-sub-9996db71-8r228\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:55.054002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.053912    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:55.225982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.225934    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:55.366758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.366723    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"]
Oct 30 12:03:55.396936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.396836    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:55.401695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.400079    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:55.552045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.552011    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca\") pod \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") "
Oct 30 12:03:55.552245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.552078    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca\") pod \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") "
Oct 30 12:03:55.552245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.552195    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r\" (UniqueName: \"kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r\") pod \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\" (UID: \"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7\") "
Oct 30 12:03:55.552499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.552422    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca" pod "2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" (UID: "2f767b3a-1fc2-4633-bdfc-feccb7e32cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:55.552817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.552797    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:55.553719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.553680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:55.578323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.578285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca" pod "2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" (UID: "2f767b3a-1fc2-4633-bdfc-feccb7e32cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:55.578323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.578303    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r" pod "2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" (UID: "2f767b3a-1fc2-4633-bdfc-feccb7e32cd7"). InnerVolumeSpecName "kube-api-access-pxs5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:55.654604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.654523    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:55.654604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:55.654551    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxs5r\" (UniqueName: \"kubernetes.io/projected/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7-kube-api-access-pxs5r\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:56.229781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:56.229065    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228" event={"ID":"f1a6f79d-a414-4a17-a47b-958653dfd58d","Type":"ContainerStarted","Data":"102551bfd2310c4ef47b3ea879bb5e12998f1e8d058f5f9e89cb364f2ead5900"}
Oct 30 12:03:56.229781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:56.229131    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228" event={"ID":"f1a6f79d-a414-4a17-a47b-958653dfd58d","Type":"ContainerStarted","Data":"43e2515dfc7fd25d60d49255419c5338f68e096fed52c7b5a50f77e838d540ed"}
Oct 30 12:03:56.230992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:56.230966    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"
Oct 30 12:03:56.231127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:56.230997    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7" event={"ID":"2f767b3a-1fc2-4633-bdfc-feccb7e32cd7","Type":"ContainerDied","Data":"b046b328e2aa0f30ba561cf674690a7ef4f65986b484c98896b495545bf63421"}
Oct 30 12:03:56.231127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:56.231048    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b046b328e2aa0f30ba561cf674690a7ef4f65986b484c98896b495545bf63421"
Oct 30 12:03:56.274805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:56.274746    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228" podStartSLOduration=2.274728199 podStartE2EDuration="2.274728199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:56.27067319 +0000 UTC m=+2040.065557684" watchObservedRunningTime="2026-10-30 12:03:56.274728199 +0000 UTC m=+2040.069612692"
Oct 30 12:03:57.242166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.242129    5060 generic.go:356] "Generic (PLEG): container finished" podID="f1a6f79d-a414-4a17-a47b-958653dfd58d" containerID="102551bfd2310c4ef47b3ea879bb5e12998f1e8d058f5f9e89cb364f2ead5900" exitCode=0
Oct 30 12:03:57.242605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.242226    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228" event={"ID":"f1a6f79d-a414-4a17-a47b-958653dfd58d","Type":"ContainerDied","Data":"102551bfd2310c4ef47b3ea879bb5e12998f1e8d058f5f9e89cb364f2ead5900"}
Oct 30 12:03:57.293051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.293012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:57.905353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.905312    5060 scope.go:122] "RemoveContainer" containerID="c9857aeca64e073e5367d2e13c718438cab96a610591216cf92075ca89bfbaa8"
Oct 30 12:03:57.917572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.917544    5060 scope.go:122] "RemoveContainer" containerID="f05574f8b0bfc5fc24ec77c41ea5fb39beb360efd7b17dcaea24484a6d7071d5"
Oct 30 12:03:57.930051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.930024    5060 scope.go:122] "RemoveContainer" containerID="df449337b3d6da617d23219279db69bb8a09ff2f653c69028527f86255675629"
Oct 30 12:03:57.940425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.940397    5060 scope.go:122] "RemoveContainer" containerID="2695d1a44ab711720d2b5c16d23015067b7e10ef4f2f65802429fa32071300e7"
Oct 30 12:03:57.950949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:57.950927    5060 scope.go:122] "RemoveContainer" containerID="f676cdd4fd736450835e0d0f8afa0ac691a66502b10dbd0c372e10f3a8c7317d"
Oct 30 12:03:58.047471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.047430    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:58.435865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.435831    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:58.599810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.599701    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l\" (UniqueName: \"kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l\") pod \"f1a6f79d-a414-4a17-a47b-958653dfd58d\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") "
Oct 30 12:03:58.600041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.599843    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca\") pod \"f1a6f79d-a414-4a17-a47b-958653dfd58d\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") "
Oct 30 12:03:58.600041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.599974    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca\") pod \"f1a6f79d-a414-4a17-a47b-958653dfd58d\" (UID: \"f1a6f79d-a414-4a17-a47b-958653dfd58d\") "
Oct 30 12:03:58.600658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.600630    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca" pod "f1a6f79d-a414-4a17-a47b-958653dfd58d" (UID: "f1a6f79d-a414-4a17-a47b-958653dfd58d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:58.602930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.602902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l" pod "f1a6f79d-a414-4a17-a47b-958653dfd58d" (UID: "f1a6f79d-a414-4a17-a47b-958653dfd58d"). InnerVolumeSpecName "kube-api-access-drs9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:58.603047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.603002    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca" pod "f1a6f79d-a414-4a17-a47b-958653dfd58d" (UID: "f1a6f79d-a414-4a17-a47b-958653dfd58d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:03:58.701546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.701505    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:58.701546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.701542    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a6f79d-a414-4a17-a47b-958653dfd58d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:58.701756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:58.701558    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drs9l\" (UniqueName: \"kubernetes.io/projected/f1a6f79d-a414-4a17-a47b-958653dfd58d-kube-api-access-drs9l\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:03:59.141831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:59.141795    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:03:59.259987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:59.259956    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"
Oct 30 12:03:59.260172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:59.260007    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228" event={"ID":"f1a6f79d-a414-4a17-a47b-958653dfd58d","Type":"ContainerDied","Data":"43e2515dfc7fd25d60d49255419c5338f68e096fed52c7b5a50f77e838d540ed"}
Oct 30 12:03:59.260172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:03:59.260049    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e2515dfc7fd25d60d49255419c5338f68e096fed52c7b5a50f77e838d540ed"
Oct 30 12:04:00.965367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:00.965315    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:02.889762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:02.889709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:03.400480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.400051    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"]
Oct 30 12:04:03.405001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.401121    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" containerName="revoke-keys"
Oct 30 12:04:03.405001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.401146    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" containerName="revoke-keys"
Oct 30 12:04:03.405001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.401200    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f1a6f79d-a414-4a17-a47b-958653dfd58d" containerName="revoke-keys"
Oct 30 12:04:03.405001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.401210    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a6f79d-a414-4a17-a47b-958653dfd58d" containerName="revoke-keys"
Oct 30 12:04:03.405001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.401475    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" containerName="revoke-keys"
Oct 30 12:04:03.405001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.401499    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a6f79d-a414-4a17-a47b-958653dfd58d" containerName="revoke-keys"
Oct 30 12:04:03.423492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.413229    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.449480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.448751    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-849669-openshift-default-dockercfg-2qnx2\""
Oct 30 12:04:03.466500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.461693    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"]
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576576    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxrn\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576636    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576706    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576731    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576759    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576789    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576819    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576879    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.577299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.576907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.686954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.686399    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.687394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.687373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.687876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.687859    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.688638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.688617    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxrn\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.689099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.689080    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.689570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.689553    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.690060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.690041    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.690486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.690465    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.697679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.697652    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.698324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.688365    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.698565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.686879    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.698689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.687800    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.698782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.691205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.698873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.689988    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.719977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.705823    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.726218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.726146    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxrn\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.733123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.733086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.750337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.750230    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:03.774983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:03.774950    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:04.298970 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:04.298902    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47eac8b2_20c1_49b3_a9e7_ff78c195a88a.slice/crio-a95fe64023112baf159215823c952fa194694f1920a8ce145ceeca1d278540a3 WatchSource:0}: Error finding container a95fe64023112baf159215823c952fa194694f1920a8ce145ceeca1d278540a3: Status 404 returned error can't find the container with id a95fe64023112baf159215823c952fa194694f1920a8ce145ceeca1d278540a3
Oct 30 12:04:04.326062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:04.324673    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"]
Oct 30 12:04:04.326062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:04.325271    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:04.326062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:04.325356    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:04.326062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:04.325399    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:04.410049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:04.410000    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" event={"ID":"47eac8b2-20c1-49b3-a9e7-ff78c195a88a","Type":"ContainerStarted","Data":"a95fe64023112baf159215823c952fa194694f1920a8ce145ceeca1d278540a3"}
Oct 30 12:04:05.307478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.307055    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:05.307478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.307217    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:04:05.317277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.316882    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:05.453199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.453146    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" event={"ID":"47eac8b2-20c1-49b3-a9e7-ff78c195a88a","Type":"ContainerStarted","Data":"392672df93de39e5a52dbb991fefe87a37593308cbbf0a5b7670dd7ed22d38fe"}
Oct 30 12:04:05.567675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.567569    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:05.783597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.778628    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:05.784005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.783979    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:05.784273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.784229    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:05.828537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:05.823539    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:06.126031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.125879    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" podUID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" containerName="main" containerID="cri-o://062532ef244b55cc5e7722f8a8143d119a1736e0845b19771362af3f5d975b1c" gracePeriod=45
Oct 30 12:04:06.170052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.170012    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" podUID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" containerName="main" probeResult="failure" output="Get \"https://10.132.0.185:8000/ready\": dial tcp 10.132.0.185:8000: connect: connection refused"
Oct 30 12:04:06.446305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.444572    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:06.463802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.463768    5060 generic.go:356] "Generic (PLEG): container finished" podID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" containerID="062532ef244b55cc5e7722f8a8143d119a1736e0845b19771362af3f5d975b1c" exitCode=0
Oct 30 12:04:06.463962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.463839    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" event={"ID":"5f9aaed8-04a4-45fe-ac86-656a21e2f422","Type":"ContainerDied","Data":"062532ef244b55cc5e7722f8a8143d119a1736e0845b19771362af3f5d975b1c"}
Oct 30 12:04:06.463962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.463892    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp" event={"ID":"5f9aaed8-04a4-45fe-ac86-656a21e2f422","Type":"ContainerDied","Data":"a8bd3ec19e8cb280d14992f05f5c1e323a4154271105f1a456a270f4541c5118"}
Oct 30 12:04:06.463962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.463917    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8bd3ec19e8cb280d14992f05f5c1e323a4154271105f1a456a270f4541c5118"
Oct 30 12:04:06.527488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.525957    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:04:06.571508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.571433    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podStartSLOduration=3.571418022 podStartE2EDuration="3.571418022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:05.557838545 +0000 UTC m=+2049.352723044" watchObservedRunningTime="2026-10-30 12:04:06.571418022 +0000 UTC m=+2050.366302516"
Oct 30 12:04:06.631053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.631012    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm\") pod \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") "
Oct 30 12:04:06.631224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.631067    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9\" (UniqueName: \"kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9\") pod \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") "
Oct 30 12:04:06.631224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.631119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir\") pod \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") "
Oct 30 12:04:06.631299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.631245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache\") pod \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") "
Oct 30 12:04:06.631338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.631316    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home\") pod \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") "
Oct 30 12:04:06.631374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.631361    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs\") pod \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\" (UID: \"5f9aaed8-04a4-45fe-ac86-656a21e2f422\") "
Oct 30 12:04:06.632436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.632404    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache" pod "5f9aaed8-04a4-45fe-ac86-656a21e2f422" (UID: "5f9aaed8-04a4-45fe-ac86-656a21e2f422"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:06.632436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.632414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home" pod "5f9aaed8-04a4-45fe-ac86-656a21e2f422" (UID: "5f9aaed8-04a4-45fe-ac86-656a21e2f422"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:06.633198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.632812    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir" pod "5f9aaed8-04a4-45fe-ac86-656a21e2f422" (UID: "5f9aaed8-04a4-45fe-ac86-656a21e2f422"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:06.634165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.634138    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs" pod "5f9aaed8-04a4-45fe-ac86-656a21e2f422" (UID: "5f9aaed8-04a4-45fe-ac86-656a21e2f422"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:04:06.634501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.634479    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm" pod "5f9aaed8-04a4-45fe-ac86-656a21e2f422" (UID: "5f9aaed8-04a4-45fe-ac86-656a21e2f422"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:06.635958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.635935    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9" pod "5f9aaed8-04a4-45fe-ac86-656a21e2f422" (UID: "5f9aaed8-04a4-45fe-ac86-656a21e2f422"). InnerVolumeSpecName "kube-api-access-vvdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:06.733265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.733173    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:06.733265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.733206    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:06.733265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.733219    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:06.733265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.733231    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvdk9\" (UniqueName: \"kubernetes.io/projected/5f9aaed8-04a4-45fe-ac86-656a21e2f422-kube-api-access-vvdk9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:06.733265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.733242    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:06.733265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.733254    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9aaed8-04a4-45fe-ac86-656a21e2f422-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:06.776309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.776272    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:06.776553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:06.776330    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:07.413539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.411179    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"]
Oct 30 12:04:07.414210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.414183    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" containerName="main"
Oct 30 12:04:07.414210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.414210    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" containerName="main"
Oct 30 12:04:07.414485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.414432    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" containerName="main"
Oct 30 12:04:07.418222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.418199    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.430289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.430258    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"]
Oct 30 12:04:07.441427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441397    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsw4\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441534    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441636    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441728    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441749    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441783    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.441946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.441933    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.472820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.472788    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"
Oct 30 12:04:07.542761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.542723    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.542927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.542770    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.542927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.542787    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.542927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.542811    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.542980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsw4\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543028    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543062    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543083    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543380    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.543963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.543719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.544133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.544028    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.544195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.544177    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.546629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.546532    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.546771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.546754    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.553343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.552886    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"]
Oct 30 12:04:07.557615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.557584    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-e25f4a/praxis-route-e25f4a-kserve-54db69b895-5hzkp"]
Oct 30 12:04:07.568847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.568816    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsw4\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.569182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.569162    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.741091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.741060    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:07.803381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.802883    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:07.803381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:07.802950    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:08.111938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.111905    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"]
Oct 30 12:04:08.115240 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:08.115204    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c50c36_46a0_45a2_a11a_643ae80f040f.slice/crio-aa9714d91620add251cf789834db2fda6965485eb112806d2d3bec7716c2a59f WatchSource:0}: Error finding container aa9714d91620add251cf789834db2fda6965485eb112806d2d3bec7716c2a59f: Status 404 returned error can't find the container with id aa9714d91620add251cf789834db2fda6965485eb112806d2d3bec7716c2a59f
Oct 30 12:04:08.118643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.118607    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:08.118755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.118693    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:08.118755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.118737    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:08.222996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.222918    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:08.338499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.336707    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9aaed8-04a4-45fe-ac86-656a21e2f422" path="/var/lib/kubelet/pods/5f9aaed8-04a4-45fe-ac86-656a21e2f422/volumes"
Oct 30 12:04:08.532074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.532036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" event={"ID":"89c50c36-46a0-45a2-a11a-643ae80f040f","Type":"ContainerStarted","Data":"577010c93663ad858926ded8eb44276425b99914487f82e9606e4bb87611c6ec"}
Oct 30 12:04:08.533131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.533106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" event={"ID":"89c50c36-46a0-45a2-a11a-643ae80f040f","Type":"ContainerStarted","Data":"aa9714d91620add251cf789834db2fda6965485eb112806d2d3bec7716c2a59f"}
Oct 30 12:04:08.621203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.620162    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podStartSLOduration=1.6201145810000002 podStartE2EDuration="1.620114581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:08.600156735 +0000 UTC m=+2052.395041230" watchObservedRunningTime="2026-10-30 12:04:08.620114581 +0000 UTC m=+2052.414999079"
Oct 30 12:04:08.741616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.741580    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:08.775799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.775757    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:08.776011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:08.775818    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:09.743596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:09.743549    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 12:04:09.744073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:09.743615    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 12:04:09.776340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:09.776300    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:09.776568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:09.776358    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:10.087578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:10.087484    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:10.741930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:10.741891    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 12:04:10.742165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:10.741952    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 12:04:10.775773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:10.775734    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:10.776224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:10.775793    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:11.742174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:11.742124    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 12:04:11.742404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:11.742179    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 12:04:11.776042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:11.776000    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 12:04:11.776443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:11.776059    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 12:04:11.893490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:11.889379    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:12.742509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:12.742443    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 12:04:12.742679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:12.742529    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 12:04:13.249478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.248872    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:13.249997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.249671    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:13.252616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.252583    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:04:13.300929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.300893    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"]
Oct 30 12:04:13.312291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.312260    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.328367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.327714    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:04:13.328367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.327714    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:04:13.328367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.327747    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:04:13.328367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.327791    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:04:13.329627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.329605    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"]
Oct 30 12:04:13.468632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.468588    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.468823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.468677    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.468934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.468906    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbwn\" (UniqueName: \"kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.571523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.570024    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.571523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.570122    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.571523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.570257    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbwn\" (UniqueName: \"kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.575487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.572058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.622505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.621761    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.624512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.624442    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbwn\" (UniqueName: \"kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn\") pod \"maas-api-revoke-sub-a3e97c23-gk7tz\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.630878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.630646    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:13.739953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.739773    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:13.745314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.744782    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 12:04:13.745314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.744840    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 12:04:13.965109 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:13.965073    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc559e6_52c2_404d_850b_0af5f67c07d7.slice/crio-718a6151bff06fd2e0b22828cee3643ea5d3f09e14553096ba0be594a3b8360d WatchSource:0}: Error finding container 718a6151bff06fd2e0b22828cee3643ea5d3f09e14553096ba0be594a3b8360d: Status 404 returned error can't find the container with id 718a6151bff06fd2e0b22828cee3643ea5d3f09e14553096ba0be594a3b8360d
Oct 30 12:04:13.969417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:13.969393    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"]
Oct 30 12:04:14.623012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:14.622922    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz" event={"ID":"5dc559e6-52c2-404d-850b-0af5f67c07d7","Type":"ContainerStarted","Data":"9d693202fe9be6099a003d2d8adaad24cd4a427eb2bd9fe2cc966e681fa2b041"}
Oct 30 12:04:14.623012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:14.622982    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz" event={"ID":"5dc559e6-52c2-404d-850b-0af5f67c07d7","Type":"ContainerStarted","Data":"718a6151bff06fd2e0b22828cee3643ea5d3f09e14553096ba0be594a3b8360d"}
Oct 30 12:04:14.668709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:14.668631    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz" podStartSLOduration=1.668610921 podStartE2EDuration="1.668610921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:14.663615079 +0000 UTC m=+2058.458499579" watchObservedRunningTime="2026-10-30 12:04:14.668610921 +0000 UTC m=+2058.463495412"
Oct 30 12:04:14.741670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:14.741622    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 12:04:14.741835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:14.741686    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 12:04:15.474751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:15.474653    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:15.635422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:15.635380    5060 generic.go:356] "Generic (PLEG): container finished" podID="5dc559e6-52c2-404d-850b-0af5f67c07d7" containerID="9d693202fe9be6099a003d2d8adaad24cd4a427eb2bd9fe2cc966e681fa2b041" exitCode=0
Oct 30 12:04:15.635869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:15.635483    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz" event={"ID":"5dc559e6-52c2-404d-850b-0af5f67c07d7","Type":"ContainerDied","Data":"9d693202fe9be6099a003d2d8adaad24cd4a427eb2bd9fe2cc966e681fa2b041"}
Oct 30 12:04:16.018602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.018563    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:16.018978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.018956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:16.019873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.019844    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:04:16.817932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.817905    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:16.935120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.935033    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn\" (UniqueName: \"kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn\") pod \"5dc559e6-52c2-404d-850b-0af5f67c07d7\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") "
Oct 30 12:04:16.935120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.935081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca\") pod \"5dc559e6-52c2-404d-850b-0af5f67c07d7\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") "
Oct 30 12:04:16.935324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.935125    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca\") pod \"5dc559e6-52c2-404d-850b-0af5f67c07d7\" (UID: \"5dc559e6-52c2-404d-850b-0af5f67c07d7\") "
Oct 30 12:04:16.935601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.935563    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca" pod "5dc559e6-52c2-404d-850b-0af5f67c07d7" (UID: "5dc559e6-52c2-404d-850b-0af5f67c07d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:16.937356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.937334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn" pod "5dc559e6-52c2-404d-850b-0af5f67c07d7" (UID: "5dc559e6-52c2-404d-850b-0af5f67c07d7"). InnerVolumeSpecName "kube-api-access-gnbwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:16.937432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.937413    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca" pod "5dc559e6-52c2-404d-850b-0af5f67c07d7" (UID: "5dc559e6-52c2-404d-850b-0af5f67c07d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:04:16.950634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:16.950597    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:17.037154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.037110    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbwn\" (UniqueName: \"kubernetes.io/projected/5dc559e6-52c2-404d-850b-0af5f67c07d7-kube-api-access-gnbwn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:17.037154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.037152    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:17.037389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.037166    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc559e6-52c2-404d-850b-0af5f67c07d7-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:17.394887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.394852    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:17.652115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.652018    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"
Oct 30 12:04:17.652115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.652027    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz" event={"ID":"5dc559e6-52c2-404d-850b-0af5f67c07d7","Type":"ContainerDied","Data":"718a6151bff06fd2e0b22828cee3643ea5d3f09e14553096ba0be594a3b8360d"}
Oct 30 12:04:17.652316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:17.652116    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718a6151bff06fd2e0b22828cee3643ea5d3f09e14553096ba0be594a3b8360d"
Oct 30 12:04:18.197580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.197541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:18.396320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.396282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:18.636485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.636373    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"]
Oct 30 12:04:18.637277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.637248    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5dc559e6-52c2-404d-850b-0af5f67c07d7" containerName="revoke-keys"
Oct 30 12:04:18.637277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.637276    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc559e6-52c2-404d-850b-0af5f67c07d7" containerName="revoke-keys"
Oct 30 12:04:18.637510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.637495    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc559e6-52c2-404d-850b-0af5f67c07d7" containerName="revoke-keys"
Oct 30 12:04:18.641892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.641865    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.666124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.666089    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"]
Oct 30 12:04:18.672023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.671998    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c3ad80-openshift-default-dockercfg-kqmfv\""
Oct 30 12:04:18.730592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.730524    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:18.760018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.759982    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760061    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760173    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760201    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760332    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5gf\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760432    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760554    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.760710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.760692    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862275    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862319    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862402    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862436    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862510    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862554    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5gf\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862685    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862757    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.862923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.862788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.863102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.863080    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.863264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.863241    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.863688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.863666    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.865019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.864997    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.865578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.865553    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.895501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.895403    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5gf\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.896889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.896861    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token\") pod \"e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") " pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:18.903602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.903396    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:18.960051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:18.959995    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:19.055479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.047409    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:19.236630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.222889    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:19.460529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.460483    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:19.485154 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:19.485111    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc01688_2286_4f28_829d_ca31b876f398.slice/crio-3d17f2ba5649466791eb443f23428f7d78bf91bc2676c4905c44fbae11e2a78d WatchSource:0}: Error finding container 3d17f2ba5649466791eb443f23428f7d78bf91bc2676c4905c44fbae11e2a78d: Status 404 returned error can't find the container with id 3d17f2ba5649466791eb443f23428f7d78bf91bc2676c4905c44fbae11e2a78d
Oct 30 12:04:19.486961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.486896    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"]
Oct 30 12:04:19.487933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.487902    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:19.488000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.487967    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:19.488000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.487995    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:19.689258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.689211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" event={"ID":"5fc01688-2286-4f28-829d-ca31b876f398","Type":"ContainerStarted","Data":"9446f42d1486d34dc2ab98b2dea902297f132a1d17efd5ab534d59bffcfc0c0f"}
Oct 30 12:04:19.689505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.689285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" event={"ID":"5fc01688-2286-4f28-829d-ca31b876f398","Type":"ContainerStarted","Data":"3d17f2ba5649466791eb443f23428f7d78bf91bc2676c4905c44fbae11e2a78d"}
Oct 30 12:04:19.742107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.741984    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podStartSLOduration=1.7419612789999999 podStartE2EDuration="1.741961279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:19.735685598 +0000 UTC m=+2063.530570093" watchObservedRunningTime="2026-10-30 12:04:19.741961279 +0000 UTC m=+2063.536845774"
Oct 30 12:04:19.897824 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:04:19.897774    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:04:19.961036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:19.960970    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:20.967547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:20.967512    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:21.090198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:21.090160    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:21.724755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:21.724715    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:21.725891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:21.725868    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:04:22.320982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.320939    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:22.489443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.489399    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"]
Oct 30 12:04:22.493880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.493850    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.494117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.494092    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:22.519747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.519708    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"]
Oct 30 12:04:22.541120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541081    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541157    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541209    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541277    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541326    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541347    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541391    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvslr\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.541861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.541420    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642407    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642446    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642494    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642528    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642544    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvslr\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642609    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642764    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.642833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642816    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.643163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.642972    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.643163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.643144    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.643280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.643200    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.643344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.643330    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.643491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.643337    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.645575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.645545    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.645694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.645648    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.670826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.670796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.670991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.670932    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvslr\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.813075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.813035    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:22.835789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.835760    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"]
Oct 30 12:04:22.840208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.840182    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.845007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.844981    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.845125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.845039    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfdn\" (UniqueName: \"kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.845174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.845152    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.860442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.860404    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:04:22.860904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.860622    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:04:22.860904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.860679    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:04:22.860904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.860685    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:04:22.865509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.865478    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"]
Oct 30 12:04:22.946942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.946902    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.947140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.946961    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfdn\" (UniqueName: \"kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.947140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.946994    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.947761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.947735    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.949792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.949767    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:22.972107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:22.972070    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfdn\" (UniqueName: \"kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn\") pod \"maas-api-revoke-sub-e72ee423-hfxqb\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:23.122607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.122572    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:23.156192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.156106    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:23.169693 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:23.169651    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7ec47f_8a5b_41fb_95e9_98096c41937b.slice/crio-84209e8e103ced9e13768088c4275b3d51cd59266f6f6b6346e1157365b8133a WatchSource:0}: Error finding container 84209e8e103ced9e13768088c4275b3d51cd59266f6f6b6346e1157365b8133a: Status 404 returned error can't find the container with id 84209e8e103ced9e13768088c4275b3d51cd59266f6f6b6346e1157365b8133a
Oct 30 12:04:23.170430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.170381    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"]
Oct 30 12:04:23.172373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.172341    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:23.172446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.172422    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:23.172548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.172515    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:23.415412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.415118    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:23.487317 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:23.487284    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e746fc_3bd6_4fe6_9d16_044c9848ec1c.slice/crio-23c54a05487cb9d22b7b943660f4272f6ad69aaec767ce7b808cbd12695840a1 WatchSource:0}: Error finding container 23c54a05487cb9d22b7b943660f4272f6ad69aaec767ce7b808cbd12695840a1: Status 404 returned error can't find the container with id 23c54a05487cb9d22b7b943660f4272f6ad69aaec767ce7b808cbd12695840a1
Oct 30 12:04:23.496859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.495211    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"]
Oct 30 12:04:23.743967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.743926    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb" event={"ID":"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c","Type":"ContainerStarted","Data":"23c54a05487cb9d22b7b943660f4272f6ad69aaec767ce7b808cbd12695840a1"}
Oct 30 12:04:23.745829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.745799    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" event={"ID":"db7ec47f-8a5b-41fb-95e9-98096c41937b","Type":"ContainerStarted","Data":"3ce51888a3239e8fd8c2be63705f25641e05804153d2cf184b14b30b5151fe31"}
Oct 30 12:04:23.745957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.745848    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" event={"ID":"db7ec47f-8a5b-41fb-95e9-98096c41937b","Type":"ContainerStarted","Data":"84209e8e103ced9e13768088c4275b3d51cd59266f6f6b6346e1157365b8133a"}
Oct 30 12:04:23.821435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.812362    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podStartSLOduration=1.8123409019999999 podStartE2EDuration="1.812340902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:23.810333591 +0000 UTC m=+2067.605218088" watchObservedRunningTime="2026-10-30 12:04:23.812340902 +0000 UTC m=+2067.607225398"
Oct 30 12:04:23.821863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:23.821786    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:24.124269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.124221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:24.304749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.304556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:24.754971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.754935    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" containerID="cce28aa5e42b92d6fd2db65fc26de9fa301ae307731c2526adfa92a585428513" exitCode=0
Oct 30 12:04:24.755407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.755084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb" event={"ID":"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c","Type":"ContainerDied","Data":"cce28aa5e42b92d6fd2db65fc26de9fa301ae307731c2526adfa92a585428513"}
Oct 30 12:04:24.763513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.763483    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:24.816096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.816060    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:24.816292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:24.816125    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:25.549942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:25.549902    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:25.825275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:25.825032    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:25.825275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:25.825087    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:25.957483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:25.955662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:26.133534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.132544    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"]
Oct 30 12:04:26.140608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.138361    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:26.148798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.148765    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"]
Oct 30 12:04:26.148987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.148898    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.165634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.165585    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f1b14f-dockercfg-g7gxs\""
Oct 30 12:04:26.165925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.165728    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"]
Oct 30 12:04:26.170367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.170340    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.198846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.175326    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f1b14f\""
Oct 30 12:04:26.223728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.221543    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"]
Oct 30 12:04:26.227910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.227438    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fss\" (UniqueName: \"kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss\") pod \"payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.227910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.227577    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.227910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.227752    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.227910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.227808    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9kc2\" (UniqueName: \"kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2\") pod \"payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.235648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.235617    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.470998    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca\") pod \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") "
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.471130    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca\") pod \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") "
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.471202    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn\" (UniqueName: \"kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn\") pod \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\" (UID: \"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c\") "
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.471359    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.471519    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.471575    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9kc2\" (UniqueName: \"kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2\") pod \"payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.486367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.471722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fss\" (UniqueName: \"kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss\") pod \"payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.523480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.519201    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.545399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.544691    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.571553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.571168    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9kc2\" (UniqueName: \"kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2\") pod \"payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:26.575003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.574958    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca" pod "c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" (UID: "c4e746fc-3bd6-4fe6-9d16-044c9848ec1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:04:26.646485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.640251    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fss\" (UniqueName: \"kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss\") pod \"payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.672937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.669857    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca" pod "c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" (UID: "c4e746fc-3bd6-4fe6-9d16-044c9848ec1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:26.672937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.672652    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn" pod "c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" (UID: "c4e746fc-3bd6-4fe6-9d16-044c9848ec1c"). InnerVolumeSpecName "kube-api-access-7lfdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:26.685710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.677287    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:26.685710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.677558    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:26.685710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.677580    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfdn\" (UniqueName: \"kubernetes.io/projected/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c-kube-api-access-7lfdn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:26.766727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.766640    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:26.776687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.776653    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb" event={"ID":"c4e746fc-3bd6-4fe6-9d16-044c9848ec1c","Type":"ContainerDied","Data":"23c54a05487cb9d22b7b943660f4272f6ad69aaec767ce7b808cbd12695840a1"}
Oct 30 12:04:26.776916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.776901    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c54a05487cb9d22b7b943660f4272f6ad69aaec767ce7b808cbd12695840a1"
Oct 30 12:04:26.777072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.777060    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"
Oct 30 12:04:26.816126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.816090    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:26.816446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.816404    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:26.838480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:26.835017    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:27.585582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:27.581019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:27.850791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:27.839201    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:27.851520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:27.851479    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:28.021468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.021400    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"]
Oct 30 12:04:28.303425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.258246    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"]
Oct 30 12:04:28.402643 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:28.402600    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f28db7_7ed3_42bd_b638_ebb40980903a.slice/crio-59477beb92850c7d2d5ce6d7acf153a19654951d754404b4f113740d4de69ef3 WatchSource:0}: Error finding container 59477beb92850c7d2d5ce6d7acf153a19654951d754404b4f113740d4de69ef3: Status 404 returned error can't find the container with id 59477beb92850c7d2d5ce6d7acf153a19654951d754404b4f113740d4de69ef3
Oct 30 12:04:28.425290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.424250    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:28.482931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.482887    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:28.872532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.859635    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:28.872532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.859737    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:28.872532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:28.860636    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 12:04:29.037944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.037897    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" event={"ID":"9fbf6674-283c-43de-810e-94f436247a5a","Type":"ContainerStarted","Data":"5c913288678e83c9452ad663fbf81c26bcba6752a281b52266825c2aac379f83"}
Oct 30 12:04:29.040222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.040131    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" event={"ID":"63f28db7-7ed3-42bd-b638-ebb40980903a","Type":"ContainerStarted","Data":"59477beb92850c7d2d5ce6d7acf153a19654951d754404b4f113740d4de69ef3"}
Oct 30 12:04:29.215732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.214938    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:29.260320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.259416    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:29.417581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.416545    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:29.549041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.547349    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:29.813423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.813388    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:29.813669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:29.813465    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:30.052051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.052006    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" event={"ID":"63f28db7-7ed3-42bd-b638-ebb40980903a","Type":"ContainerStarted","Data":"a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8"}
Oct 30 12:04:30.052555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.052538    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:30.054142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.054113    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" event={"ID":"9fbf6674-283c-43de-810e-94f436247a5a","Type":"ContainerStarted","Data":"916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8"}
Oct 30 12:04:30.054262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.054182    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:30.101821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.101688    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" podStartSLOduration=4.101664531 podStartE2EDuration="4.101664531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:30.097834922 +0000 UTC m=+2073.892719419" watchObservedRunningTime="2026-10-30 12:04:30.101664531 +0000 UTC m=+2073.896549028"
Oct 30 12:04:30.139717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.139662    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" podStartSLOduration=4.139647277 podStartE2EDuration="4.139647277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:30.138583373 +0000 UTC m=+2073.933467868" watchObservedRunningTime="2026-10-30 12:04:30.139647277 +0000 UTC m=+2073.934531818"
Oct 30 12:04:30.330440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.330410    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:30.814105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.814056    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 12:04:30.814295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:30.814127    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 12:04:31.869438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:31.869405    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:31.869984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:31.869906    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:31.871174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:31.871148    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:04:32.145533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:32.145258    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:33.253889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.253856    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"]
Oct 30 12:04:33.254425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.254412    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" containerName="revoke-keys"
Oct 30 12:04:33.254425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.254426    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" containerName="revoke-keys"
Oct 30 12:04:33.254619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.254606    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" containerName="revoke-keys"
Oct 30 12:04:33.258540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.258515    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.280657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.280624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"]
Oct 30 12:04:33.284588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.284358    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:04:33.284588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.284419    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:04:33.284588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.284558    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:04:33.286026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.285999    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:04:33.303746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.303708    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwg8s\" (UniqueName: \"kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.303953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.303825    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.303953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.303919    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.405466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.405409    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.405682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.405664    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwg8s\" (UniqueName: \"kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.405782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.405766    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.406247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.406220    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.408270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.408249    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.429859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.429826    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwg8s\" (UniqueName: \"kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s\") pod \"maas-api-revoke-sub-a355c016-97pkn\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.574629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.574542    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:33.890150 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:33.889919    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a81023_9ef9_4246_955a_1cfec4895f55.slice/crio-660c647785b5e6af5724d5bdf9c93d3b0c4afcf6a70f772632cb2a7c103f6a56 WatchSource:0}: Error finding container 660c647785b5e6af5724d5bdf9c93d3b0c4afcf6a70f772632cb2a7c103f6a56: Status 404 returned error can't find the container with id 660c647785b5e6af5724d5bdf9c93d3b0c4afcf6a70f772632cb2a7c103f6a56
Oct 30 12:04:33.893355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.893330    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"]
Oct 30 12:04:33.953638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:33.953604    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:34.094663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:34.094623    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn" event={"ID":"42a81023-9ef9-4246-955a-1cfec4895f55","Type":"ContainerStarted","Data":"660c647785b5e6af5724d5bdf9c93d3b0c4afcf6a70f772632cb2a7c103f6a56"}
Oct 30 12:04:35.128485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.128362    5060 generic.go:356] "Generic (PLEG): container finished" podID="42a81023-9ef9-4246-955a-1cfec4895f55" containerID="1b0585ce6c7df719e457e0ca6de8e3c687166ec17a109860edd093cf1e471c0b" exitCode=0
Oct 30 12:04:35.128995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.128617    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn" event={"ID":"42a81023-9ef9-4246-955a-1cfec4895f55","Type":"ContainerDied","Data":"1b0585ce6c7df719e457e0ca6de8e3c687166ec17a109860edd093cf1e471c0b"}
Oct 30 12:04:35.264965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.264917    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:35.422539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.419003    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:35.664009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.663967    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:35.827557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.827521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:35.991183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.991140    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"]
Oct 30 12:04:35.991893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:35.991463    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy" containerID="cri-o://3cadf045920aac80e7f67d4b67d82219336dd5bffcc38d05d52e4c82183653bf" gracePeriod=30
Oct 30 12:04:36.000729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.000692    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"]
Oct 30 12:04:36.000974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.000947    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy" containerID="cri-o://fce1ab6a75f2197dfed3638e756ed3158b3e6840ac75870a72632bc264f28ccc" gracePeriod=30
Oct 30 12:04:36.003817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.003781    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:36.012129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.012094    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"]
Oct 30 12:04:36.012374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.012340    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy" containerID="cri-o://1733dc5dd1f8afca9f911cb5ce0ea71662e32f860081365df3009bd4289e9788" gracePeriod=30
Oct 30 12:04:36.167587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.166310    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:36.360645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.360617    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:36.459815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.459774    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca\") pod \"42a81023-9ef9-4246-955a-1cfec4895f55\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") "
Oct 30 12:04:36.460008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.459840    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s\" (UniqueName: \"kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s\") pod \"42a81023-9ef9-4246-955a-1cfec4895f55\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") "
Oct 30 12:04:36.460055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.460003    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca\") pod \"42a81023-9ef9-4246-955a-1cfec4895f55\" (UID: \"42a81023-9ef9-4246-955a-1cfec4895f55\") "
Oct 30 12:04:36.460436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.460399    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca" pod "42a81023-9ef9-4246-955a-1cfec4895f55" (UID: "42a81023-9ef9-4246-955a-1cfec4895f55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:36.461045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.461021    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:36.462323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.462298    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca" pod "42a81023-9ef9-4246-955a-1cfec4895f55" (UID: "42a81023-9ef9-4246-955a-1cfec4895f55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:04:36.462554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.462531    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s" pod "42a81023-9ef9-4246-955a-1cfec4895f55" (UID: "42a81023-9ef9-4246-955a-1cfec4895f55"). InnerVolumeSpecName "kube-api-access-wwg8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:36.562379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.562345    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a81023-9ef9-4246-955a-1cfec4895f55-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:36.562379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.562377    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwg8s\" (UniqueName: \"kubernetes.io/projected/42a81023-9ef9-4246-955a-1cfec4895f55-kube-api-access-wwg8s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:36.670615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.670577    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"]
Oct 30 12:04:36.671567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.671543    5060 cpu_manager.go:414] "containerMap: removing container" podUID="42a81023-9ef9-4246-955a-1cfec4895f55" containerName="revoke-keys"
Oct 30 12:04:36.671656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.671569    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a81023-9ef9-4246-955a-1cfec4895f55" containerName="revoke-keys"
Oct 30 12:04:36.671770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.671759    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a81023-9ef9-4246-955a-1cfec4895f55" containerName="revoke-keys"
Oct 30 12:04:36.676163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.676142    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.707040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.705958    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"]
Oct 30 12:04:36.764075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.763980    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxkp\" (UniqueName: \"kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.764075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.764038    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.764301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.764118    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.865353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.865314    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxkp\" (UniqueName: \"kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.865596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.865380    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.865596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.865422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.866225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.866200    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.868188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.868161    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:36.895119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:36.895073    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxkp\" (UniqueName: \"kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp\") pod \"maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:37.003199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:37.003159    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:37.188239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:37.188196    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn" event={"ID":"42a81023-9ef9-4246-955a-1cfec4895f55","Type":"ContainerDied","Data":"660c647785b5e6af5724d5bdf9c93d3b0c4afcf6a70f772632cb2a7c103f6a56"}
Oct 30 12:04:37.188239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:37.188222    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"
Oct 30 12:04:37.188864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:37.188265    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660c647785b5e6af5724d5bdf9c93d3b0c4afcf6a70f772632cb2a7c103f6a56"
Oct 30 12:04:37.344387 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:37.344358    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c28b97_0d07_4b25_91f5_d4fa09075886.slice/crio-3b2640a7978978c7e6e575df374951e7b811ff128dbadae307aa24281bf8f3a6 WatchSource:0}: Error finding container 3b2640a7978978c7e6e575df374951e7b811ff128dbadae307aa24281bf8f3a6: Status 404 returned error can't find the container with id 3b2640a7978978c7e6e575df374951e7b811ff128dbadae307aa24281bf8f3a6
Oct 30 12:04:37.346959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:37.346830    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"]
Oct 30 12:04:38.227475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:38.224691    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt" event={"ID":"e1c28b97-0d07-4b25-91f5-d4fa09075886","Type":"ContainerStarted","Data":"2cd48f93180d6dd15a30d8ca2fb82ee88e96d339658bcc6535015f14a2de58fc"}
Oct 30 12:04:38.227475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:38.224825    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt" event={"ID":"e1c28b97-0d07-4b25-91f5-d4fa09075886","Type":"ContainerStarted","Data":"3b2640a7978978c7e6e575df374951e7b811ff128dbadae307aa24281bf8f3a6"}
Oct 30 12:04:38.271315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:38.271261    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt" podStartSLOduration=2.271248335 podStartE2EDuration="2.271248335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:38.27036585 +0000 UTC m=+2082.065250337" watchObservedRunningTime="2026-10-30 12:04:38.271248335 +0000 UTC m=+2082.066132829"
Oct 30 12:04:38.301524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:38.301468    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:38.469479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:38.468441    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:39.241174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:39.241134    5060 generic.go:356] "Generic (PLEG): container finished" podID="e1c28b97-0d07-4b25-91f5-d4fa09075886" containerID="2cd48f93180d6dd15a30d8ca2fb82ee88e96d339658bcc6535015f14a2de58fc" exitCode=0
Oct 30 12:04:39.241718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:39.241407    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt" event={"ID":"e1c28b97-0d07-4b25-91f5-d4fa09075886","Type":"ContainerDied","Data":"2cd48f93180d6dd15a30d8ca2fb82ee88e96d339658bcc6535015f14a2de58fc"}
Oct 30 12:04:39.486200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:39.486138    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:39.690568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:39.685467    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:40.425331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.425304    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:40.537901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.537804    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca\") pod \"e1c28b97-0d07-4b25-91f5-d4fa09075886\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") "
Oct 30 12:04:40.538052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.537934    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca\") pod \"e1c28b97-0d07-4b25-91f5-d4fa09075886\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") "
Oct 30 12:04:40.538105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.538090    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp\" (UniqueName: \"kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp\") pod \"e1c28b97-0d07-4b25-91f5-d4fa09075886\" (UID: \"e1c28b97-0d07-4b25-91f5-d4fa09075886\") "
Oct 30 12:04:40.538275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.538252    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca" pod "e1c28b97-0d07-4b25-91f5-d4fa09075886" (UID: "e1c28b97-0d07-4b25-91f5-d4fa09075886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:40.538585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.538569    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:40.540122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.540097    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca" pod "e1c28b97-0d07-4b25-91f5-d4fa09075886" (UID: "e1c28b97-0d07-4b25-91f5-d4fa09075886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:04:40.540317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.540296    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp" pod "e1c28b97-0d07-4b25-91f5-d4fa09075886" (UID: "e1c28b97-0d07-4b25-91f5-d4fa09075886"). InnerVolumeSpecName "kube-api-access-hrxkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:40.640331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.640294    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxkp\" (UniqueName: \"kubernetes.io/projected/e1c28b97-0d07-4b25-91f5-d4fa09075886-kube-api-access-hrxkp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:40.640331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.640328    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c28b97-0d07-4b25-91f5-d4fa09075886-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:40.796984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:40.796878    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:41.065727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.065635    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:04:41.065727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.065712    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:04:41.265722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.265523    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"
Oct 30 12:04:41.265722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.265520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt" event={"ID":"e1c28b97-0d07-4b25-91f5-d4fa09075886","Type":"ContainerDied","Data":"3b2640a7978978c7e6e575df374951e7b811ff128dbadae307aa24281bf8f3a6"}
Oct 30 12:04:41.265722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.265650    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2640a7978978c7e6e575df374951e7b811ff128dbadae307aa24281bf8f3a6"
Oct 30 12:04:41.275473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.275412    5060 generic.go:356] "Generic (PLEG): container finished" podID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerID="3cadf045920aac80e7f67d4b67d82219336dd5bffcc38d05d52e4c82183653bf" exitCode=0
Oct 30 12:04:41.275683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.275540    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" event={"ID":"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5","Type":"ContainerDied","Data":"3cadf045920aac80e7f67d4b67d82219336dd5bffcc38d05d52e4c82183653bf"}
Oct 30 12:04:41.275683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.275601    5060 scope.go:122] "RemoveContainer" containerID="5aef57c1ff5eac80ca5883b31cfeca72c410655664574ae1f7a656d5186cd277"
Oct 30 12:04:41.281221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.281163    5060 generic.go:356] "Generic (PLEG): container finished" podID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerID="1733dc5dd1f8afca9f911cb5ce0ea71662e32f860081365df3009bd4289e9788" exitCode=0
Oct 30 12:04:41.281433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.281405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" event={"ID":"eb048634-eb56-4b15-9cd0-cc65cecf3a62","Type":"ContainerDied","Data":"1733dc5dd1f8afca9f911cb5ce0ea71662e32f860081365df3009bd4289e9788"}
Oct 30 12:04:41.285675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.285642    5060 generic.go:356] "Generic (PLEG): container finished" podID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerID="fce1ab6a75f2197dfed3638e756ed3158b3e6840ac75870a72632bc264f28ccc" exitCode=0
Oct 30 12:04:41.285845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.285824    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" event={"ID":"c396b539-68c8-44d1-88e8-45117a4f7ea9","Type":"ContainerDied","Data":"fce1ab6a75f2197dfed3638e756ed3158b3e6840ac75870a72632bc264f28ccc"}
Oct 30 12:04:41.508525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.507699    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:04:41.511291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.511267    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 12:04:41.515012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.514987    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 12:04:41.558859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558573    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.558859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558620    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.558859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558674    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.558859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558710    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.558859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558795    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.558859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558830    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558903    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.558935    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559024    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559084    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559128    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559162    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559187    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559274    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559268    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559345    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k\") pod \"c396b539-68c8-44d1-88e8-45117a4f7ea9\" (UID: \"c396b539-68c8-44d1-88e8-45117a4f7ea9\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559403    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.559835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.559480    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket\") pod \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\" (UID: \"eb048634-eb56-4b15-9cd0-cc65cecf3a62\") "
Oct 30 12:04:41.560576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.560110    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.560576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.560362    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.561368    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.561401    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.561721    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.561893    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.562154    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.562205    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:41.562364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.562201    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.563359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.563315    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "kube-api-access-64f9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.564184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.564156    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.565245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.565210    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.565577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.565555    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.570866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.570557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.570866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.570557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.570866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.570689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:04:41.572346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.572309    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo" pod "eb048634-eb56-4b15-9cd0-cc65cecf3a62" (UID: "eb048634-eb56-4b15-9cd0-cc65cecf3a62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:04:41.572624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.572594    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k" pod "c396b539-68c8-44d1-88e8-45117a4f7ea9" (UID: "c396b539-68c8-44d1-88e8-45117a4f7ea9"). InnerVolumeSpecName "kube-api-access-z527k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.660555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660523    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.660761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660628    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.660761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660655    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.660761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660749    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.660952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.660952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660926    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.661067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.660997    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.661067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661045    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.661172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket\") pod \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\" (UID: \"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5\") "
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661513    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661537    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c396b539-68c8-44d1-88e8-45117a4f7ea9-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661551    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661562    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661575    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z527k\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-kube-api-access-z527k\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661589    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661601    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661615    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64f9q\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-kube-api-access-64f9q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661627    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661640    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661656    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661667    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661679    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661691    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661703    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb048634-eb56-4b15-9cd0-cc65cecf3a62-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661715    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.661970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.661728    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c396b539-68c8-44d1-88e8-45117a4f7ea9-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.663050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.662072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.664832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.664659    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.665474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.665153    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:41.665883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.665629    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.666826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.666794    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.670150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.670116    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:04:41.670150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.670126    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.670150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.670123    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.672020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.671993    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5" pod "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" (UID: "2fb80fc4-9c8d-43d5-9ca7-27b07da634a5"). InnerVolumeSpecName "kube-api-access-xb5k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.699923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.699863    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:41.762796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762760    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5k5\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-kube-api-access-xb5k5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.762796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762797    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762813    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762826    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762839    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762851    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762866    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762879    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.763024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.762891    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.855069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.854986    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"]
Oct 30 12:04:41.855657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855629    5060 cpu_manager.go:414] "containerMap: removing container" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy"
Oct 30 12:04:41.855657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855656    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855710    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855719    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855743    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855751    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855774    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e1c28b97-0d07-4b25-91f5-d4fa09075886" containerName="revoke-keys"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855783    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c28b97-0d07-4b25-91f5-d4fa09075886" containerName="revoke-keys"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855800    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy"
Oct 30 12:04:41.855845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855806    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy"
Oct 30 12:04:41.856133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855946    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" containerName="istio-proxy"
Oct 30 12:04:41.856133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855957    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy"
Oct 30 12:04:41.856133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855966    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c28b97-0d07-4b25-91f5-d4fa09075886" containerName="revoke-keys"
Oct 30 12:04:41.856133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855972    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" containerName="istio-proxy"
Oct 30 12:04:41.856133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.855978    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" containerName="istio-proxy"
Oct 30 12:04:41.860763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.860745    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:41.877951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.877881    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-28ca05\""
Oct 30 12:04:41.878208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.878180    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-28ca05-dockercfg-7r8vq\""
Oct 30 12:04:41.882330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.882174    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:41.883951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.883929    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"]
Oct 30 12:04:41.999543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.977524    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:41.999543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:41.977744    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwtp\" (UniqueName: \"kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp\") pod \"payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:42.044029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.044000    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"]
Oct 30 12:04:42.049390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.049348    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.057416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.057352    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"]
Oct 30 12:04:42.079613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.079572    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwtp\" (UniqueName: \"kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp\") pod \"payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:42.079818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.079679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:42.080500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.080476    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:42.108082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.107994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwtp\" (UniqueName: \"kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp\") pod \"payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:42.176724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.176689    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:42.180666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.180639    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.180967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.180937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgbf\" (UniqueName: \"kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf\") pod \"payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.281932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.281889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.282183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.282162    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgbf\" (UniqueName: \"kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf\") pod \"payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.283129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.283104    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.309189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.309150    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgbf\" (UniqueName: \"kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf\") pod \"payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.310527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.310496    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"
Oct 30 12:04:42.310685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.310594    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc" event={"ID":"2fb80fc4-9c8d-43d5-9ca7-27b07da634a5","Type":"ContainerDied","Data":"111b68c615fe5a07e7cacb2d3d961c9527cf10e0cbd802ca6dbd5d55f4fe6bbf"}
Oct 30 12:04:42.310685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.310661    5060 scope.go:122] "RemoveContainer" containerID="3cadf045920aac80e7f67d4b67d82219336dd5bffcc38d05d52e4c82183653bf"
Oct 30 12:04:42.318207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.318184    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"
Oct 30 12:04:42.319239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.318202    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b" event={"ID":"eb048634-eb56-4b15-9cd0-cc65cecf3a62","Type":"ContainerDied","Data":"d8cb7e7b467b958289c2e090ae49affdefdf9e678d1c8c83d2b8007b3b4eaa45"}
Oct 30 12:04:42.321080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.320939    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"
Oct 30 12:04:42.321719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.321690    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv" event={"ID":"c396b539-68c8-44d1-88e8-45117a4f7ea9","Type":"ContainerDied","Data":"74d6c9f2a0af24610aaff6fe9848d07e25450cae9e14c01cd3b68bd78bd5df41"}
Oct 30 12:04:42.328638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.328608    5060 scope.go:122] "RemoveContainer" containerID="1733dc5dd1f8afca9f911cb5ce0ea71662e32f860081365df3009bd4289e9788"
Oct 30 12:04:42.344877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.344855    5060 scope.go:122] "RemoveContainer" containerID="fce1ab6a75f2197dfed3638e756ed3158b3e6840ac75870a72632bc264f28ccc"
Oct 30 12:04:42.372272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.372183    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"]
Oct 30 12:04:42.373892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.373865    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:42.378787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.378768    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-jxm7b"]
Oct 30 12:04:42.422149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.422109    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"]
Oct 30 12:04:42.427215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.427179    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-m75vc"]
Oct 30 12:04:42.483001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.482965    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"]
Oct 30 12:04:42.493950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.493885    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4pcgv"]
Oct 30 12:04:42.494624 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:42.494600    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca546bf_94a7_4db4_81fe_c3208ba6223f.slice/crio-76cb793ccfdd02c24c29b224f86085585885f98a106da164b2513ea555f8935d WatchSource:0}: Error finding container 76cb793ccfdd02c24c29b224f86085585885f98a106da164b2513ea555f8935d: Status 404 returned error can't find the container with id 76cb793ccfdd02c24c29b224f86085585885f98a106da164b2513ea555f8935d
Oct 30 12:04:42.501183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.500915    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"]
Oct 30 12:04:42.696358 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:42.696320    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb8e3d4_f91b_4820_9e40_9ccbfdca7b0e.slice/crio-0ae6eea3f3f1e14e18fb120a3fdc3fd8ffbfa07b63dfbc9cc20001470134c06d WatchSource:0}: Error finding container 0ae6eea3f3f1e14e18fb120a3fdc3fd8ffbfa07b63dfbc9cc20001470134c06d: Status 404 returned error can't find the container with id 0ae6eea3f3f1e14e18fb120a3fdc3fd8ffbfa07b63dfbc9cc20001470134c06d
Oct 30 12:04:42.698691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.698666    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"]
Oct 30 12:04:42.857717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:42.857672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:43.013259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.013225    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:43.351848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.351732    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78" event={"ID":"1ca546bf-94a7-4db4-81fe-c3208ba6223f","Type":"ContainerStarted","Data":"abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63"}
Oct 30 12:04:43.352056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.351914    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78" event={"ID":"1ca546bf-94a7-4db4-81fe-c3208ba6223f","Type":"ContainerStarted","Data":"76cb793ccfdd02c24c29b224f86085585885f98a106da164b2513ea555f8935d"}
Oct 30 12:04:43.352203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.352168    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:43.354434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.354407    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c" event={"ID":"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e","Type":"ContainerStarted","Data":"f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964"}
Oct 30 12:04:43.354552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.354476    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c" event={"ID":"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e","Type":"ContainerStarted","Data":"0ae6eea3f3f1e14e18fb120a3fdc3fd8ffbfa07b63dfbc9cc20001470134c06d"}
Oct 30 12:04:43.354552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.354527    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:43.424756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.424697    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78" podStartSLOduration=2.424679699 podStartE2EDuration="2.424679699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:43.423079915 +0000 UTC m=+2087.217964409" watchObservedRunningTime="2026-10-30 12:04:43.424679699 +0000 UTC m=+2087.219564192"
Oct 30 12:04:43.496776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:43.496723    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c" podStartSLOduration=2.4967074670000002 podStartE2EDuration="2.496707467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:43.496130334 +0000 UTC m=+2087.291014828" watchObservedRunningTime="2026-10-30 12:04:43.496707467 +0000 UTC m=+2087.291591961"
Oct 30 12:04:44.339440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:44.339406    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb80fc4-9c8d-43d5-9ca7-27b07da634a5" path="/var/lib/kubelet/pods/2fb80fc4-9c8d-43d5-9ca7-27b07da634a5/volumes"
Oct 30 12:04:44.340065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:44.340035    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c396b539-68c8-44d1-88e8-45117a4f7ea9" path="/var/lib/kubelet/pods/c396b539-68c8-44d1-88e8-45117a4f7ea9/volumes"
Oct 30 12:04:44.340520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:44.340504    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb048634-eb56-4b15-9cd0-cc65cecf3a62" path="/var/lib/kubelet/pods/eb048634-eb56-4b15-9cd0-cc65cecf3a62/volumes"
Oct 30 12:04:45.145028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:45.144988    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:47.080966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:47.080859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:47.179766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:47.179721    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"]
Oct 30 12:04:47.370184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:47.370086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:47.520322 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:47.520281    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:48.500167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.500125    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"]
Oct 30 12:04:48.506860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.506831    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.519255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.519226    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"]
Oct 30 12:04:48.524955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.524912    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.529579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.529547    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"]
Oct 30 12:04:48.550303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.550265    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"]
Oct 30 12:04:48.674026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.673986    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674028    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674065    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674153    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvxs\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674188    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrwf\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674207    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674302    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674344    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674370    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674416    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674542    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.674706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674585    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674615    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674703    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674763    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.674870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674822    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.675009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674869    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.675009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.674948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776091    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776141    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776167    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776206    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776245    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776307    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvxs\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776352    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrwf\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776410    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776437    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776502    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776535    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776558    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776598    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776603    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776691    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776734    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776751    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776794    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776825    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776903    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.776990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.776992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.777703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.777305    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.777703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.777342    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.777703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.777552    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.778040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.777778    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.778109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.778087    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.778769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.778742    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.779289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.779268    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.779591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.779560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.779882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.779860    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.780157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.780140    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.800605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.800569    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.800830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.800807    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.801111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.801089    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvxs\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:48.816570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.816531    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrwf\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.825299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.825268    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:48.844248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:48.844215    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:49.030690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.030581    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:49.160577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.158806    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"]
Oct 30 12:04:49.176592 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:49.176548    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4fcd11_c4ed_4bfd_b448_cef2afb67d78.slice/crio-9bb5186a3a75aabb4fe4e5aa4131589a1ef19c13d3ad338c8f1a111533959a3f WatchSource:0}: Error finding container 9bb5186a3a75aabb4fe4e5aa4131589a1ef19c13d3ad338c8f1a111533959a3f: Status 404 returned error can't find the container with id 9bb5186a3a75aabb4fe4e5aa4131589a1ef19c13d3ad338c8f1a111533959a3f
Oct 30 12:04:49.179597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.179556    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:49.179713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.179655    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:49.179713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.179699    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:49.198134 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:04:49.198105    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1481ed_d4fe_4ee8_b254_fb666f62bd93.slice/crio-d1b120013684e87e3ffddc557b24739ca4c4a3edd8cf52d99133c76a3c685ed5 WatchSource:0}: Error finding container d1b120013684e87e3ffddc557b24739ca4c4a3edd8cf52d99133c76a3c685ed5: Status 404 returned error can't find the container with id d1b120013684e87e3ffddc557b24739ca4c4a3edd8cf52d99133c76a3c685ed5
Oct 30 12:04:49.201382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.201360    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"]
Oct 30 12:04:49.201582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.201545    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:49.201658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.201628    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:49.201717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.201668    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:04:49.415407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.415311    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" event={"ID":"cf1481ed-d4fe-4ee8-b254-fb666f62bd93","Type":"ContainerStarted","Data":"a8a53dce4d3739d97bc2c550a24981534ac35d65d2090cd6bd5f957d131dcd5d"}
Oct 30 12:04:49.415407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.415380    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" event={"ID":"cf1481ed-d4fe-4ee8-b254-fb666f62bd93","Type":"ContainerStarted","Data":"d1b120013684e87e3ffddc557b24739ca4c4a3edd8cf52d99133c76a3c685ed5"}
Oct 30 12:04:49.418657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.418622    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" event={"ID":"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78","Type":"ContainerStarted","Data":"437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c"}
Oct 30 12:04:49.418825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.418683    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" event={"ID":"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78","Type":"ContainerStarted","Data":"9bb5186a3a75aabb4fe4e5aa4131589a1ef19c13d3ad338c8f1a111533959a3f"}
Oct 30 12:04:49.464734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.464673    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podStartSLOduration=1.464655942 podStartE2EDuration="1.464655942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:49.462995485 +0000 UTC m=+2093.257879980" watchObservedRunningTime="2026-10-30 12:04:49.464655942 +0000 UTC m=+2093.259540435"
Oct 30 12:04:49.514985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.512545    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podStartSLOduration=1.512524301 podStartE2EDuration="1.512524301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:49.509763689 +0000 UTC m=+2093.304648183" watchObservedRunningTime="2026-10-30 12:04:49.512524301 +0000 UTC m=+2093.307408798"
Oct 30 12:04:49.826235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.826202    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:49.845044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:49.844999    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:04:49.858574 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:04:49.858537    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:04:50.766795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:50.766751    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:50.848476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:50.841309    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:50.848476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:50.841378    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:50.854474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:50.852569    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:50.854474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:50.852641    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:50.973393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:50.973329    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:51.829480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:51.828932    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:51.829480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:51.828993    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:51.845932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:51.845889    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:51.846094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:51.845951    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:52.201476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.198088    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj" podUID="17a97f6f-bbc0-456c-a3bc-2b80c75ba406" containerName="payload-pre-processing" containerID="cri-o://b9b053191770076807a5e0c9d227351de4f3fa6465ec84635701ed8a0adc35bb" gracePeriod=25
Oct 30 12:04:52.465529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.464847    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:52.598355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.598317    5060 generic.go:356] "Generic (PLEG): container finished" podID="17a97f6f-bbc0-456c-a3bc-2b80c75ba406" containerID="b9b053191770076807a5e0c9d227351de4f3fa6465ec84635701ed8a0adc35bb" exitCode=0
Oct 30 12:04:52.598936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.598903    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj" event={"ID":"17a97f6f-bbc0-456c-a3bc-2b80c75ba406","Type":"ContainerDied","Data":"b9b053191770076807a5e0c9d227351de4f3fa6465ec84635701ed8a0adc35bb"}
Oct 30 12:04:52.826975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.826879    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:52.827259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.827229    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:52.895484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.889931    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:52.895484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:52.890002    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:53.049507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.048809    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:53.149758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.149708    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"]
Oct 30 12:04:53.158481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.158443    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-eb5531-hgbrt"]
Oct 30 12:04:53.176022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.175997    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:04:53.225256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.223677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:53.263633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.259909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume\") pod \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") "
Oct 30 12:04:53.263633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.260042    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k\" (UniqueName: \"kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k\") pod \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\" (UID: \"17a97f6f-bbc0-456c-a3bc-2b80c75ba406\") "
Oct 30 12:04:53.263633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.261384    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume" pod "17a97f6f-bbc0-456c-a3bc-2b80c75ba406" (UID: "17a97f6f-bbc0-456c-a3bc-2b80c75ba406"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:53.276281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.276242    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k" pod "17a97f6f-bbc0-456c-a3bc-2b80c75ba406" (UID: "17a97f6f-bbc0-456c-a3bc-2b80c75ba406"). InnerVolumeSpecName "kube-api-access-zf28k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:53.363301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.363252    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:53.363301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.363298    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf28k\" (UniqueName: \"kubernetes.io/projected/17a97f6f-bbc0-456c-a3bc-2b80c75ba406-kube-api-access-zf28k\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:04:53.370987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.370953    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:53.642298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.642257    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj" event={"ID":"17a97f6f-bbc0-456c-a3bc-2b80c75ba406","Type":"ContainerDied","Data":"656dfad0dce588c12df76235a181659b78a654c52e64d465c82046b259df5d4d"}
Oct 30 12:04:53.642520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.642353    5060 scope.go:122] "RemoveContainer" containerID="b9b053191770076807a5e0c9d227351de4f3fa6465ec84635701ed8a0adc35bb"
Oct 30 12:04:53.642595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.642584    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"
Oct 30 12:04:53.674717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.674679    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:53.727612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.727529    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"]
Oct 30 12:04:53.746101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.746060    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb5531-7fb4dbcf9-qsggj"]
Oct 30 12:04:53.830477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.829873    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:53.830477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.829938    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:53.846477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.845017    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:53.846477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:53.845073    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:54.348851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.348820    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a97f6f-bbc0-456c-a3bc-2b80c75ba406" path="/var/lib/kubelet/pods/17a97f6f-bbc0-456c-a3bc-2b80c75ba406/volumes"
Oct 30 12:04:54.349362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.349273    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c28b97-0d07-4b25-91f5-d4fa09075886" path="/var/lib/kubelet/pods/e1c28b97-0d07-4b25-91f5-d4fa09075886/volumes"
Oct 30 12:04:54.365475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.365414    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:04:54.365679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.365529    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:04:54.826936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.826893    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:54.827241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.827211    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:54.844821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.844723    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:54.844821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:54.844773    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:55.392480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:55.392144    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:55.825927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:55.825887    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:55.826237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:55.826204    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:55.847766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:55.847719    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:55.848125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:55.848086    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:56.831478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:56.830902    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 12:04:56.831478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:56.830963    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 12:04:56.857478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:56.850526    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 12:04:56.857478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:56.850585    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 12:04:57.226944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:57.226911    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:04:57.261705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:57.261673    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:04:57.940481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:57.914963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:58.151765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.151728    5060 scope.go:122] "RemoveContainer" containerID="4c814fae7d9445e989dbfb0e22e95a2b75b350641f6f8cf9af4d0c22f574c138"
Oct 30 12:04:58.201539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.196688    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:04:58.294927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.294888    5060 scope.go:122] "RemoveContainer" containerID="b05469dcded8de5801dd54814628e670feb9893cbe29e0c070ff36e89df1dbf4"
Oct 30 12:04:58.326500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.324968    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:58.350102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.350044    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:58.452539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.422688    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:58.648493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.646612    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:58.697624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.697568    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:58.872298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.871567    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:04:58.872298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.871651    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:58.890212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.889589    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:04:58.890212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:58.889649    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:59.412879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:59.412846    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:59.414882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:59.414859    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:59.424902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:59.424826    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:04:59.846192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:59.846148    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:04:59.896477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:59.895305    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:04:59.896477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:04:59.895366    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:05:00.066609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.064904    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:05:00.066609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.066539    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:05:00.085522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.071058    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:05:00.161932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.161839    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:00.718022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.717982    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"]
Oct 30 12:05:00.718784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.718761    5060 cpu_manager.go:414] "containerMap: removing container" podUID="17a97f6f-bbc0-456c-a3bc-2b80c75ba406" containerName="payload-pre-processing"
Oct 30 12:05:00.718883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.718786    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a97f6f-bbc0-456c-a3bc-2b80c75ba406" containerName="payload-pre-processing"
Oct 30 12:05:00.719004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.718989    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a97f6f-bbc0-456c-a3bc-2b80c75ba406" containerName="payload-pre-processing"
Oct 30 12:05:00.724847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.724821    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.742049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.742020    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:05:00.744307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.744275    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"]
Oct 30 12:05:00.751661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.751636    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:05:00.751808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.751677    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:05:00.751808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.751735    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:05:00.775643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.775590    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59tc\" (UniqueName: \"kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.775826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.775793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.776023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.775952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.896967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.896924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59tc\" (UniqueName: \"kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.897380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.897360    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.898381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.898269    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.898742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.898719    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.908539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.908441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.922022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.921989    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59tc\" (UniqueName: \"kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc\") pod \"maas-api-revoke-sub-5dfec3b4-c6b6s\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:00.943438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:00.943398    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:01.039586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:01.039495    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:01.175388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:01.175354    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:01.336378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:01.335828    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"]
Oct 30 12:05:01.336378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:01.336069    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy" containerID="cri-o://b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e" gracePeriod=30
Oct 30 12:05:01.465476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:01.454515    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"]
Oct 30 12:05:01.519515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:01.515491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:02.356246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:02.356209    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s" event={"ID":"5b777693-bd41-4fab-9cc1-137588803863","Type":"ContainerStarted","Data":"5eefcc6934746ef0907a8c28191e1155ba0d69473a3eed484175711d7700a644"}
Oct 30 12:05:02.356743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:02.356269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s" event={"ID":"5b777693-bd41-4fab-9cc1-137588803863","Type":"ContainerStarted","Data":"d5c52ecd1dd7e59c51d8b54b069e830a4cb25b83b0dff7a27aeffadc751e5dd7"}
Oct 30 12:05:02.430482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:02.416782    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:03.367790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:03.367756    5060 generic.go:356] "Generic (PLEG): container finished" podID="5b777693-bd41-4fab-9cc1-137588803863" containerID="5eefcc6934746ef0907a8c28191e1155ba0d69473a3eed484175711d7700a644" exitCode=0
Oct 30 12:05:03.368215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:03.367821    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s" event={"ID":"5b777693-bd41-4fab-9cc1-137588803863","Type":"ContainerDied","Data":"5eefcc6934746ef0907a8c28191e1155ba0d69473a3eed484175711d7700a644"}
Oct 30 12:05:04.103758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.103683    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"]
Oct 30 12:05:04.109509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.109472    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.135470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.135418    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"]
Oct 30 12:05:04.213562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.213518    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.213791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.213595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjbc\" (UniqueName: \"kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.213791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.213755    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.315631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.315594    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.316221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.315878    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjbc\" (UniqueName: \"kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.316221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.316010    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.316939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.316915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.319198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.319171    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.340030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.339991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjbc\" (UniqueName: \"kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc\") pod \"maas-api-revoke-sub-65649434-h5j6l\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.428867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.428835    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:04.586860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.586822    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:04.628229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.628086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:04.725757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.725661    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca\") pod \"5b777693-bd41-4fab-9cc1-137588803863\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") "
Oct 30 12:05:04.725931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.725812    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca\") pod \"5b777693-bd41-4fab-9cc1-137588803863\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") "
Oct 30 12:05:04.725931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.725867    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc\" (UniqueName: \"kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc\") pod \"5b777693-bd41-4fab-9cc1-137588803863\" (UID: \"5b777693-bd41-4fab-9cc1-137588803863\") "
Oct 30 12:05:04.726227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.726200    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca" pod "5b777693-bd41-4fab-9cc1-137588803863" (UID: "5b777693-bd41-4fab-9cc1-137588803863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:05:04.726505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.726480    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b777693-bd41-4fab-9cc1-137588803863-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:04.728273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.728246    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc" pod "5b777693-bd41-4fab-9cc1-137588803863" (UID: "5b777693-bd41-4fab-9cc1-137588803863"). InnerVolumeSpecName "kube-api-access-t59tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:05:04.728422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.728404    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca" pod "5b777693-bd41-4fab-9cc1-137588803863" (UID: "5b777693-bd41-4fab-9cc1-137588803863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:05:04.789777 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:04.789740    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e4d31d_9879_4630_a4cc_3e5c00f578dc.slice/crio-a8e611cd4e3fb6d0e023d32e51f7242a0ff292994fd6bbc35ef5f2a8417361b5 WatchSource:0}: Error finding container a8e611cd4e3fb6d0e023d32e51f7242a0ff292994fd6bbc35ef5f2a8417361b5: Status 404 returned error can't find the container with id a8e611cd4e3fb6d0e023d32e51f7242a0ff292994fd6bbc35ef5f2a8417361b5
Oct 30 12:05:04.792648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.792621    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"]
Oct 30 12:05:04.827626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.827595    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t59tc\" (UniqueName: \"kubernetes.io/projected/5b777693-bd41-4fab-9cc1-137588803863-kube-api-access-t59tc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:04.827759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:04.827629    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b777693-bd41-4fab-9cc1-137588803863-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:05.389673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:05.389580    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l" event={"ID":"36e4d31d-9879-4630-a4cc-3e5c00f578dc","Type":"ContainerStarted","Data":"ee4a6f31aaf1b5dac17c541527d9c31107b24cbc0b2eb46a5b68f2a27ba645ba"}
Oct 30 12:05:05.389673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:05.389655    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l" event={"ID":"36e4d31d-9879-4630-a4cc-3e5c00f578dc","Type":"ContainerStarted","Data":"a8e611cd4e3fb6d0e023d32e51f7242a0ff292994fd6bbc35ef5f2a8417361b5"}
Oct 30 12:05:05.392588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:05.392554    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s" event={"ID":"5b777693-bd41-4fab-9cc1-137588803863","Type":"ContainerDied","Data":"d5c52ecd1dd7e59c51d8b54b069e830a4cb25b83b0dff7a27aeffadc751e5dd7"}
Oct 30 12:05:05.392769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:05.392655    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c52ecd1dd7e59c51d8b54b069e830a4cb25b83b0dff7a27aeffadc751e5dd7"
Oct 30 12:05:05.392769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:05.392717    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"
Oct 30 12:05:05.441764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:05.441705    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l" podStartSLOduration=1.441691774 podStartE2EDuration="1.441691774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:05.437679568 +0000 UTC m=+2109.232564061" watchObservedRunningTime="2026-10-30 12:05:05.441691774 +0000 UTC m=+2109.236576268"
Oct 30 12:05:06.407007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.406968    5060 generic.go:356] "Generic (PLEG): container finished" podID="36e4d31d-9879-4630-a4cc-3e5c00f578dc" containerID="ee4a6f31aaf1b5dac17c541527d9c31107b24cbc0b2eb46a5b68f2a27ba645ba" exitCode=0
Oct 30 12:05:06.407206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.407050    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l" event={"ID":"36e4d31d-9879-4630-a4cc-3e5c00f578dc","Type":"ContainerDied","Data":"ee4a6f31aaf1b5dac17c541527d9c31107b24cbc0b2eb46a5b68f2a27ba645ba"}
Oct 30 12:05:06.665775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.665747    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 12:05:06.760642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760596    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.760844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760730    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.760844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760770    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.760844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760810    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.761024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760877    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.761024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760922    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.761024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760925    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:05:06.761024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.760971    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.761223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.761023    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.761223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.761084    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo\") pod \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\" (UID: \"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0\") "
Oct 30 12:05:06.761772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.761711    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.761772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.761759    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:05:06.762137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.761901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:05:06.762137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.761964    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:05:06.762399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.762374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:05:06.764653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.764630    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:05:06.764765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.764643    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:05:06.764765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.764646    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:05:06.765780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.765758    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg" pod "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" (UID: "3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0"). InnerVolumeSpecName "kube-api-access-wtldg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:05:06.863633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863601    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863630    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863644    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863655    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863664    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863672    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtldg\" (UniqueName: \"kubernetes.io/projected/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-kube-api-access-wtldg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863683    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.863896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.863703    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:06.864758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:06.864727    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:07.418813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.418774    5060 generic.go:356] "Generic (PLEG): container finished" podID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerID="b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e" exitCode=0
Oct 30 12:05:07.419024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.418881    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"
Oct 30 12:05:07.419024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.418911    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" event={"ID":"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0","Type":"ContainerDied","Data":"b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e"}
Oct 30 12:05:07.419024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.418967    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78" event={"ID":"3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0","Type":"ContainerDied","Data":"b83a7d8c728b89c4b2961b0c94eff3146ac0c4e6575ea93923507c21081bed89"}
Oct 30 12:05:07.419024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.418991    5060 scope.go:122] "RemoveContainer" containerID="b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e"
Oct 30 12:05:07.435147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.435127    5060 scope.go:122] "RemoveContainer" containerID="dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"
Oct 30 12:05:07.449351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.449311    5060 scope.go:122] "RemoveContainer" containerID="b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e"
Oct 30 12:05:07.449851 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:05:07.449825    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e\": container with ID starting with b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e not found: ID does not exist" containerID="b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e"
Oct 30 12:05:07.449956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.449879    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e"} err="failed to get container status \"b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e\": rpc error: code = NotFound desc = could not find container \"b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e\": container with ID starting with b89f3cf6b990cfe9534449412fe601c4d27e15bf09a64d2100bf6bd727d38f7e not found: ID does not exist"
Oct 30 12:05:07.449956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.449910    5060 scope.go:122] "RemoveContainer" containerID="dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"
Oct 30 12:05:07.450285 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:05:07.450261    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1\": container with ID starting with dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1 not found: ID does not exist" containerID="dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"
Oct 30 12:05:07.450367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.450291    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1"} err="failed to get container status \"dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1\": rpc error: code = NotFound desc = could not find container \"dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1\": container with ID starting with dae46336617311ed49aa82ec90f502a7f62dac04c1027d0f3ec2cc53b5ca52f1 not found: ID does not exist"
Oct 30 12:05:07.552582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.552545    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"]
Oct 30 12:05:07.584029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.583996    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb5531-openshift-default-57bf5b5d65-bhs78"]
Oct 30 12:05:07.599235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.599204    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:07.675071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.674993    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca\") pod \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") "
Oct 30 12:05:07.675487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.675153    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca\") pod \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") "
Oct 30 12:05:07.675487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.675288    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc\" (UniqueName: \"kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc\") pod \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\" (UID: \"36e4d31d-9879-4630-a4cc-3e5c00f578dc\") "
Oct 30 12:05:07.675487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.675396    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca" pod "36e4d31d-9879-4630-a4cc-3e5c00f578dc" (UID: "36e4d31d-9879-4630-a4cc-3e5c00f578dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:05:07.675736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.675719    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:07.677595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.677557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca" pod "36e4d31d-9879-4630-a4cc-3e5c00f578dc" (UID: "36e4d31d-9879-4630-a4cc-3e5c00f578dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:05:07.677710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.677667    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc" pod "36e4d31d-9879-4630-a4cc-3e5c00f578dc" (UID: "36e4d31d-9879-4630-a4cc-3e5c00f578dc"). InnerVolumeSpecName "kube-api-access-tzjbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:05:07.766703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.766659    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:05:07.766876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.766724    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:05:07.777047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.777013    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e4d31d-9879-4630-a4cc-3e5c00f578dc-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:07.777047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:07.777045    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjbc\" (UniqueName: \"kubernetes.io/projected/36e4d31d-9879-4630-a4cc-3e5c00f578dc-kube-api-access-tzjbc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:08.115262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.115219    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:08.336002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.335927    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" path="/var/lib/kubelet/pods/3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0/volumes"
Oct 30 12:05:08.431534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.431433    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"
Oct 30 12:05:08.431534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.431481    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l" event={"ID":"36e4d31d-9879-4630-a4cc-3e5c00f578dc","Type":"ContainerDied","Data":"a8e611cd4e3fb6d0e023d32e51f7242a0ff292994fd6bbc35ef5f2a8417361b5"}
Oct 30 12:05:08.431752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.431538    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8e611cd4e3fb6d0e023d32e51f7242a0ff292994fd6bbc35ef5f2a8417361b5"
Oct 30 12:05:08.434878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.434852    5060 generic.go:356] "Generic (PLEG): container finished" podID="5fc01688-2286-4f28-829d-ca31b876f398" containerID="9446f42d1486d34dc2ab98b2dea902297f132a1d17efd5ab534d59bffcfc0c0f" exitCode=0
Oct 30 12:05:08.435032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.434893    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" event={"ID":"5fc01688-2286-4f28-829d-ca31b876f398","Type":"ContainerDied","Data":"9446f42d1486d34dc2ab98b2dea902297f132a1d17efd5ab534d59bffcfc0c0f"}
Oct 30 12:05:08.435279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.435266    5060 scope.go:122] "RemoveContainer" containerID="9446f42d1486d34dc2ab98b2dea902297f132a1d17efd5ab534d59bffcfc0c0f"
Oct 30 12:05:08.436771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.436746    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:08.436834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.436811    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:08.436870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.436841    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:08.921265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:08.921167    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:09.450865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:09.450823    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" event={"ID":"5fc01688-2286-4f28-829d-ca31b876f398","Type":"ContainerStarted","Data":"23910df04bdbce9f827e7543666c9e541720f7c6467a40a97b1e094257605f50"}
Oct 30 12:05:09.597481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:09.595928    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:09.961115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:09.961082    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:05:09.961534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:09.961362    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 12:05:09.961534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:09.961418    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 12:05:10.503480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.502629    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"]
Oct 30 12:05:10.507914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.507882    5060 cpu_manager.go:414] "containerMap: removing container" podUID="36e4d31d-9879-4630-a4cc-3e5c00f578dc" containerName="revoke-keys"
Oct 30 12:05:10.507914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.507913    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e4d31d-9879-4630-a4cc-3e5c00f578dc" containerName="revoke-keys"
Oct 30 12:05:10.508157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.507946    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5b777693-bd41-4fab-9cc1-137588803863" containerName="revoke-keys"
Oct 30 12:05:10.508157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.507954    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b777693-bd41-4fab-9cc1-137588803863" containerName="revoke-keys"
Oct 30 12:05:10.508157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.507978    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy"
Oct 30 12:05:10.508157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.507986    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy"
Oct 30 12:05:10.508371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.508315    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e4d31d-9879-4630-a4cc-3e5c00f578dc" containerName="revoke-keys"
Oct 30 12:05:10.508371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.508349    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b777693-bd41-4fab-9cc1-137588803863" containerName="revoke-keys"
Oct 30 12:05:10.508371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.508364    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy"
Oct 30 12:05:10.508566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.508376    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy"
Oct 30 12:05:10.512543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.512516    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.523229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.523195    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"]
Oct 30 12:05:10.528990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.528960    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:05:10.529312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.529288    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:05:10.529404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.529357    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:05:10.540177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.540151    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:05:10.620356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.620318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.620660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.620640    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.620896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.620885    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rt4\" (UniqueName: \"kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.722682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.722634    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rt4\" (UniqueName: \"kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.722889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.722744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.722889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.722790    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.723879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.723785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.725646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.725624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.747394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.747371    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rt4\" (UniqueName: \"kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4\") pod \"maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.831959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.831674    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:10.960985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.960950    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 12:05:10.961160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:10.961012    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 12:05:11.171234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:11.171205    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"]
Oct 30 12:05:11.172238 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:11.172211    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd02ac7_de6a_4816_9c7c_ad4ca75224df.slice/crio-4836c550c1674fbc52c81684c33392271e3337092089c385deeb22c11bd30250 WatchSource:0}: Error finding container 4836c550c1674fbc52c81684c33392271e3337092089c385deeb22c11bd30250: Status 404 returned error can't find the container with id 4836c550c1674fbc52c81684c33392271e3337092089c385deeb22c11bd30250
Oct 30 12:05:11.470131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:11.470100    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl" event={"ID":"abd02ac7-de6a-4816-9c7c-ad4ca75224df","Type":"ContainerStarted","Data":"4836c550c1674fbc52c81684c33392271e3337092089c385deeb22c11bd30250"}
Oct 30 12:05:11.721858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:11.721762    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:11.870488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:11.870433    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:11.960801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:11.960764    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 12:05:11.961038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:11.960815    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 12:05:12.168002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.167910    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"]
Oct 30 12:05:12.168844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.168817    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy"
Oct 30 12:05:12.168844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.168842    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dda6dab-3eed-443f-9a4a-b5d7e3fd3de0" containerName="istio-proxy"
Oct 30 12:05:12.173291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.173271    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.214486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.207773    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-933195-openshift-default-dockercfg-thdsv\""
Oct 30 12:05:12.217568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.217539    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"]
Oct 30 12:05:12.244292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244262    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244539    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bx87\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244630    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244664    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244706    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244780    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244850    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.244912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244897    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.245016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.244952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.284186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.284147    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:12.444521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.396238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.444521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.397037    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.444521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.443882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.444521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.443964    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bx87\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.444915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.444702    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.444915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.444845    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.444915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.444903    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.479474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.445120    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.479474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.445511    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.479474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.446331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.479474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.449685    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.495474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.487869    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.495953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.495922    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.496150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.496134    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.505519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.501037    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.512523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.512492    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.544248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.541148    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.549313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.548642    5060 generic.go:356] "Generic (PLEG): container finished" podID="abd02ac7-de6a-4816-9c7c-ad4ca75224df" containerID="4f693454f7f0b5be44e6547acef8e65cb06f31e3798c430783d400c8cbfb5684" exitCode=0
Oct 30 12:05:12.549313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.548842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl" event={"ID":"abd02ac7-de6a-4816-9c7c-ad4ca75224df","Type":"ContainerDied","Data":"4f693454f7f0b5be44e6547acef8e65cb06f31e3798c430783d400c8cbfb5684"}
Oct 30 12:05:12.550602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.550571    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bx87\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87\") pod \"e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") " pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.610949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.610905    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:12.833765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.833733    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:12.960692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.960661    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 12:05:12.960971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:12.960944    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 12:05:13.213965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.213917    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"]
Oct 30 12:05:13.218632 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:13.216941    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a395608_0ff6_42de_adb2_c425bae17902.slice/crio-d26bc7488912ed458d9572434cc29e01eeafbc395bed3e6d76aae865e4b8c050 WatchSource:0}: Error finding container d26bc7488912ed458d9572434cc29e01eeafbc395bed3e6d76aae865e4b8c050: Status 404 returned error can't find the container with id d26bc7488912ed458d9572434cc29e01eeafbc395bed3e6d76aae865e4b8c050
Oct 30 12:05:13.219949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.219911    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:13.220060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.219985    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:13.220060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.220026    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:13.565616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.565532    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" event={"ID":"7a395608-0ff6-42de-adb2-c425bae17902","Type":"ContainerStarted","Data":"d7ed3f075a69da122994c708abda8d8759ecf6cbea893dffb5abff5bd62ef319"}
Oct 30 12:05:13.565957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.565933    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" event={"ID":"7a395608-0ff6-42de-adb2-c425bae17902","Type":"ContainerStarted","Data":"d26bc7488912ed458d9572434cc29e01eeafbc395bed3e6d76aae865e4b8c050"}
Oct 30 12:05:13.631476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.631180    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podStartSLOduration=1.631142284 podStartE2EDuration="1.631142284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:13.629018575 +0000 UTC m=+2117.423903064" watchObservedRunningTime="2026-10-30 12:05:13.631142284 +0000 UTC m=+2117.426026782"
Oct 30 12:05:13.864556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.864446    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:13.904235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.904124    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:13.956527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.955411    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:13.960846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.960813    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 12:05:13.960979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:13.960878    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 12:05:14.066801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.066770    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca\") pod \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") "
Oct 30 12:05:14.066994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.066868    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4\" (UniqueName: \"kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4\") pod \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") "
Oct 30 12:05:14.066994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.066914    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca\") pod \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\" (UID: \"abd02ac7-de6a-4816-9c7c-ad4ca75224df\") "
Oct 30 12:05:14.067870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.067841    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca" pod "abd02ac7-de6a-4816-9c7c-ad4ca75224df" (UID: "abd02ac7-de6a-4816-9c7c-ad4ca75224df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:05:14.070198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.070168    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4" pod "abd02ac7-de6a-4816-9c7c-ad4ca75224df" (UID: "abd02ac7-de6a-4816-9c7c-ad4ca75224df"). InnerVolumeSpecName "kube-api-access-k9rt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:05:14.070304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.070231    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca" pod "abd02ac7-de6a-4816-9c7c-ad4ca75224df" (UID: "abd02ac7-de6a-4816-9c7c-ad4ca75224df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:05:14.168841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.168761    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:14.169009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.168998    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9rt4\" (UniqueName: \"kubernetes.io/projected/abd02ac7-de6a-4816-9c7c-ad4ca75224df-kube-api-access-k9rt4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:14.169081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.169072    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd02ac7-de6a-4816-9c7c-ad4ca75224df-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:05:14.577544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.577507    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl" event={"ID":"abd02ac7-de6a-4816-9c7c-ad4ca75224df","Type":"ContainerDied","Data":"4836c550c1674fbc52c81684c33392271e3337092089c385deeb22c11bd30250"}
Oct 30 12:05:14.578064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.577570    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4836c550c1674fbc52c81684c33392271e3337092089c385deeb22c11bd30250"
Oct 30 12:05:14.578064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.577643    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"
Oct 30 12:05:14.840158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.840089    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:14.965434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.965403    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:05:14.976219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.976162    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:05:14.977399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:14.977376    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:05:15.585596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:15.585568    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:15.586649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:15.586622    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:05:15.742150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:15.742111    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:16.017064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.017021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:16.236508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.232788    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"]
Oct 30 12:05:16.236508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.233704    5060 cpu_manager.go:414] "containerMap: removing container" podUID="abd02ac7-de6a-4816-9c7c-ad4ca75224df" containerName="revoke-keys"
Oct 30 12:05:16.236508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.233722    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd02ac7-de6a-4816-9c7c-ad4ca75224df" containerName="revoke-keys"
Oct 30 12:05:16.236508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.233867    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd02ac7-de6a-4816-9c7c-ad4ca75224df" containerName="revoke-keys"
Oct 30 12:05:16.238504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.238479    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.273390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.273297    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-87f2e6-openshift-default-dockercfg-g9p4m\""
Oct 30 12:05:16.290493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.289884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.292920    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"]
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294442    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294548    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294597    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294629    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294733    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294751    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.298020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.294809    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r276x\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396359    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r276x\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396447    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396503    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396552    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396638    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396794    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396820    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.396976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.396865    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.397262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.397224    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.397327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.397262    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.397954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.397905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.398473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.398352    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.401250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.401214    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.401375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.401263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.424826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.424789    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r276x\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.427298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.427266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token\") pod \"e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") " pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.496477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.493405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:16.568530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.568157    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:16.963305 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:16.963274    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb182d45_f45a_46c8_a9b4_0d2d97113b72.slice/crio-997dbda2dc5754df4fe47ac56a05c6454f63d5355f3dcae3df6657a138095540 WatchSource:0}: Error finding container 997dbda2dc5754df4fe47ac56a05c6454f63d5355f3dcae3df6657a138095540: Status 404 returned error can't find the container with id 997dbda2dc5754df4fe47ac56a05c6454f63d5355f3dcae3df6657a138095540
Oct 30 12:05:16.964981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.964959    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"]
Oct 30 12:05:16.966365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.966330    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:16.966493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.966412    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:16.966493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:16.966476    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:17.640528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:17.640490    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" event={"ID":"cb182d45-f45a-46c8-a9b4-0d2d97113b72","Type":"ContainerStarted","Data":"d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"}
Oct 30 12:05:17.640776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:17.640541    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" event={"ID":"cb182d45-f45a-46c8-a9b4-0d2d97113b72","Type":"ContainerStarted","Data":"997dbda2dc5754df4fe47ac56a05c6454f63d5355f3dcae3df6657a138095540"}
Oct 30 12:05:17.714827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:17.714771    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podStartSLOduration=1.714756637 podStartE2EDuration="1.714756637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:17.698273191 +0000 UTC m=+2121.493157685" watchObservedRunningTime="2026-10-30 12:05:17.714756637 +0000 UTC m=+2121.509641131"
Oct 30 12:05:18.116605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.116561    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:18.217974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.217935    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"]
Oct 30 12:05:18.224126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.224097    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.239051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.239024    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"]
Oct 30 12:05:18.242168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.242144    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-933195-dockercfg-nh8jb\""
Oct 30 12:05:18.243925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.243900    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-933195\""
Oct 30 12:05:18.355065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.349102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwl29\" (UniqueName: \"kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29\") pod \"payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.355065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.349162    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.391530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.384829    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:18.488126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.488087    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwl29\" (UniqueName: \"kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29\") pod \"payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.488682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.488652    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.494033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.493993    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.495741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.495704    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"]
Oct 30 12:05:18.504689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.503117    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"]
Oct 30 12:05:18.504689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.503382    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.528194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.527838    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwl29\" (UniqueName: \"kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29\") pod \"payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.543129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.543100    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:18.569688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.569662    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:18.577298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.577267    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:18.588040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.587939    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:18.655401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.654916    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:18.656700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.656369    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:05:18.693165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.693133    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.693500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.693480    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdnp\" (UniqueName: \"kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp\") pod \"payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.795515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.795474    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.795730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.795545    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdnp\" (UniqueName: \"kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp\") pod \"payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.796278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.796256    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.819932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.819900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdnp\" (UniqueName: \"kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp\") pod \"payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.832290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.827955    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:18.909196 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:18.891595    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87de9ccf_f4be_4360_a5f4_bafa41fd8dbd.slice/crio-2301182ab9c44b3110defb1e1774f6ecfb8314c149b4ec719d0079b47cb5e25b WatchSource:0}: Error finding container 2301182ab9c44b3110defb1e1774f6ecfb8314c149b4ec719d0079b47cb5e25b: Status 404 returned error can't find the container with id 2301182ab9c44b3110defb1e1774f6ecfb8314c149b4ec719d0079b47cb5e25b
Oct 30 12:05:18.931707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:18.927903    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"]
Oct 30 12:05:19.144656 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:19.144626    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e244ec_eeec_4b46_bd38_392aeac08f65.slice/crio-02f68a57ee69060ce995c465c0925f7bbf60b7b2b3de000b016cebf3a98fd3f8 WatchSource:0}: Error finding container 02f68a57ee69060ce995c465c0925f7bbf60b7b2b3de000b016cebf3a98fd3f8: Status 404 returned error can't find the container with id 02f68a57ee69060ce995c465c0925f7bbf60b7b2b3de000b016cebf3a98fd3f8
Oct 30 12:05:19.147585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.147560    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"]
Oct 30 12:05:19.428408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.428316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:19.666660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.666594    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" event={"ID":"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd","Type":"ContainerStarted","Data":"5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f"}
Oct 30 12:05:19.666660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.666650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" event={"ID":"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd","Type":"ContainerStarted","Data":"2301182ab9c44b3110defb1e1774f6ecfb8314c149b4ec719d0079b47cb5e25b"}
Oct 30 12:05:19.666923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.666676    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:19.668548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.668526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" event={"ID":"a8e244ec-eeec-4b46-bd38-392aeac08f65","Type":"ContainerStarted","Data":"e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec"}
Oct 30 12:05:19.668671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.668564    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" event={"ID":"a8e244ec-eeec-4b46-bd38-392aeac08f65","Type":"ContainerStarted","Data":"02f68a57ee69060ce995c465c0925f7bbf60b7b2b3de000b016cebf3a98fd3f8"}
Oct 30 12:05:19.714685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.714625    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" podStartSLOduration=1.714606088 podStartE2EDuration="1.714606088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:19.708417273 +0000 UTC m=+2123.503301767" watchObservedRunningTime="2026-10-30 12:05:19.714606088 +0000 UTC m=+2123.509490590"
Oct 30 12:05:19.772423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:19.772369    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" podStartSLOduration=1.772353275 podStartE2EDuration="1.772353275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:19.770782993 +0000 UTC m=+2123.565667487" watchObservedRunningTime="2026-10-30 12:05:19.772353275 +0000 UTC m=+2123.567237806"
Oct 30 12:05:19.858350 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:05:19.858309    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:05:20.018430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:20.018316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:20.400205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:20.400098    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:20.587283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:20.587245    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:20.679094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:20.678997    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:21.017900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:21.017480    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:21.306801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:21.306668    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:21.505594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:21.505554    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:22.612857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:22.612819    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:22.850551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:22.850508    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:23.697614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:23.697581    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:24.299273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:24.299219    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:24.483295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:24.483259    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:25.355355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:25.355316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:25.823378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:25.823342    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:26.135564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.135445    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:26.281190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.281151    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"]
Oct 30 12:05:26.286849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.286824    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.313182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.313145    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"]
Oct 30 12:05:26.341975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.341430    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:26.356872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.355188    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b02580-openshift-default-dockercfg-w2sb5\""
Oct 30 12:05:26.361556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.360687    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.369818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.369933    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.369971    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.370051    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.370109    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.370158    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.370231    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzj42\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.370279    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.372529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.370319    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.384168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.384144    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4fff4f\""
Oct 30 12:05:26.392021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.391949    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"]
Oct 30 12:05:26.392021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.392007    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"]
Oct 30 12:05:26.398009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.397981    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4fff4f-dockercfg-bjmbv\""
Oct 30 12:05:26.513477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.472489    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.514245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.514212    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.514518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.514499    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.514733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.514715    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.514878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.514863    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.518477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.515034    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzj42\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.556477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.555513    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.556477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.555965    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.556477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.556028    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgmtj\" (UniqueName: \"kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj\") pod \"payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.556477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.556099    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.556477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.556184    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.613483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.519009    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.613483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.501494    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.613483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.560033    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.613483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.519228    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.613483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.548828    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:26.613483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.560668    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.628482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.626958    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.660109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.641001    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.660109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.649802    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.660109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.651137    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzj42\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42\") pod \"e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") " pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.660109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.659390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.665348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.664120    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgmtj\" (UniqueName: \"kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj\") pod \"payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.665348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.664839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.693037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.693003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgmtj\" (UniqueName: \"kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj\") pod \"payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:26.710445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.710411    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:26.949511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.949472    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:26.991007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:26.990698    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:27.287627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.287533    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"]
Oct 30 12:05:27.292062 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:27.292026    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a37c5b7_b5a1_4389_baa2_692562ae1b96.slice/crio-793e344d32d16f42cbdb8937fe831e730627e3b858710a49004fa4bb2e750d8a WatchSource:0}: Error finding container 793e344d32d16f42cbdb8937fe831e730627e3b858710a49004fa4bb2e750d8a: Status 404 returned error can't find the container with id 793e344d32d16f42cbdb8937fe831e730627e3b858710a49004fa4bb2e750d8a
Oct 30 12:05:27.295711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.295674    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:27.295810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.295758    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:27.295810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.295798    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:27.329368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.329339    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"]
Oct 30 12:05:27.755698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.755660    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" event={"ID":"2f3063e9-89dc-4de4-8412-575d2d6778d5","Type":"ContainerStarted","Data":"821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244"}
Oct 30 12:05:27.756530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.755723    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" event={"ID":"2f3063e9-89dc-4de4-8412-575d2d6778d5","Type":"ContainerStarted","Data":"5b0bae501430942864aff3f412490e468aa5bc507708aabd477c304d33b9c3b6"}
Oct 30 12:05:27.756530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.755752    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:27.758387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.758364    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" event={"ID":"9a37c5b7-b5a1-4389-baa2-692562ae1b96","Type":"ContainerStarted","Data":"d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"}
Oct 30 12:05:27.758546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.758411    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" event={"ID":"9a37c5b7-b5a1-4389-baa2-692562ae1b96","Type":"ContainerStarted","Data":"793e344d32d16f42cbdb8937fe831e730627e3b858710a49004fa4bb2e750d8a"}
Oct 30 12:05:27.802407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.800650    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" podStartSLOduration=1.800630618 podStartE2EDuration="1.800630618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:27.79737174 +0000 UTC m=+2131.592256235" watchObservedRunningTime="2026-10-30 12:05:27.800630618 +0000 UTC m=+2131.595515113"
Oct 30 12:05:27.846762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.846712    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podStartSLOduration=1.846696721 podStartE2EDuration="1.846696721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:27.843983426 +0000 UTC m=+2131.638867919" watchObservedRunningTime="2026-10-30 12:05:27.846696721 +0000 UTC m=+2131.641581215"
Oct 30 12:05:27.949617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:27.949582    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:28.537488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:28.537409    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:28.955967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:28.955939    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:29.774135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:29.774109    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:29.775113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:29.775093    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:05:30.329241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:30.329205    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:30.530296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:30.530255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:30.682651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:30.682557    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:05:30.698283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:30.698246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:30.883041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:30.883005    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:31.691578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:31.691544    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:05:32.043135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:32.043100    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"]
Oct 30 12:05:32.047605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:32.047576    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb7ec7c-vnbhk"]
Oct 30 12:05:32.341691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:32.341513    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:32.351168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:32.351135    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f72832-c1d3-4030-a059-c04ebd1450a2" path="/var/lib/kubelet/pods/63f72832-c1d3-4030-a059-c04ebd1450a2/volumes"
Oct 30 12:05:32.531702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:32.531664    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:33.190112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:33.190077    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:34.787135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:34.787099    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:35.461646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.461597    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"]
Oct 30 12:05:35.467552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.467527    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.470665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.470638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:35.492437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.492403    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"]
Oct 30 12:05:35.497613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.497583    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-28ca05-openshift-default-dockercfg-9kwzs\""
Oct 30 12:05:35.504373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504339    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.504585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504427    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.504585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504516    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.504738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504669    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkqc\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.504803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.504864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504829    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.504930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504919    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.505015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.504990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.505175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.505150    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.606860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.606826    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.606860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.606868    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.606885    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.606942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.606965    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607034    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkqc\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607108    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607168    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607380    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607427    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607665    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.607821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.607720    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.609661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.609638    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.609984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.609966    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.631500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.631472    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkqc\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.635934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.635909    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.639288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.639265    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"]
Oct 30 12:05:35.649348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.649322    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9c1994-b2zvl"]
Oct 30 12:05:35.664963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.664924    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:35.785334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.785243    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:35.843066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:35.843031    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:36.174547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.174511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:36.175053 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:36.175028    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf592eb9a_921f_4d16_91e6_63e9dd925b9e.slice/crio-ea727f04a00c1098926746584a3c015bd6d5caa09845c1fa8ce499a88bc591b1 WatchSource:0}: Error finding container ea727f04a00c1098926746584a3c015bd6d5caa09845c1fa8ce499a88bc591b1: Status 404 returned error can't find the container with id ea727f04a00c1098926746584a3c015bd6d5caa09845c1fa8ce499a88bc591b1
Oct 30 12:05:36.177825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.177801    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"]
Oct 30 12:05:36.177973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.177946    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:36.178022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.178009    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:36.178060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.178045    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:05:36.338081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.338047    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd02ac7-de6a-4816-9c7c-ad4ca75224df" path="/var/lib/kubelet/pods/abd02ac7-de6a-4816-9c7c-ad4ca75224df/volumes"
Oct 30 12:05:36.859345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.859306    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" event={"ID":"f592eb9a-921f-4d16-91e6-63e9dd925b9e","Type":"ContainerStarted","Data":"9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90"}
Oct 30 12:05:36.859803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.859396    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" event={"ID":"f592eb9a-921f-4d16-91e6-63e9dd925b9e","Type":"ContainerStarted","Data":"ea727f04a00c1098926746584a3c015bd6d5caa09845c1fa8ce499a88bc591b1"}
Oct 30 12:05:36.944115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:36.944058    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podStartSLOduration=1.9440451250000002 podStartE2EDuration="1.944045125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:36.92959169 +0000 UTC m=+2140.724476181" watchObservedRunningTime="2026-10-30 12:05:36.944045125 +0000 UTC m=+2140.738929618"
Oct 30 12:05:37.801701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:37.801667    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:37.803304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:37.803276    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 12:05:37.803462 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:37.803328    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 12:05:37.808819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:37.808788    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:37.963038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:37.962993    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:38.129967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:38.129887    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:38.771665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:38.771634    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:05:38.800480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:38.791598    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 12:05:38.800480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:38.791651    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 12:05:39.789479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:39.785887    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 12:05:39.789479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:39.785946    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 12:05:40.009338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:40.009305    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:40.791478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:40.790910    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 12:05:40.791478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:40.790996    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 12:05:41.799479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:41.787900    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 12:05:41.799479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:41.787991    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 12:05:42.087083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:42.086989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:42.784673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:42.784634    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:42.808985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:42.808953    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:42.810441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:42.810419    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:42.815503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:42.811602    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:05:42.995503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:42.993814    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:44.012159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:44.012123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:44.190039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:44.190000    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:46.051299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:46.051261    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:48.026838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:48.026800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:48.189033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:48.188990    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:49.857406 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:05:49.857370    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:05:49.902730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:49.902627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:51.123669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:51.123622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:51.279879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:51.279841    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:51.670755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:51.670719    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:53.165403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:53.165279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:53.644760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:53.644710    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:55.432703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:55.432665    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:55.584541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:55.584489    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:55.766972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:55.766933    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:57.487077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:57.487040    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:05:58.704059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.704014    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"]
Oct 30 12:05:58.710633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.710610    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.724970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.724943    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"]
Oct 30 12:05:58.732061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.731859    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:05:58.732061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.731969    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:05:58.732061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.732034    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:05:58.737875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.737844    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:05:58.830364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.830328    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.830590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.830470    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdk9\" (UniqueName: \"kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.830590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.830514    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.905761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.905729    5060 scope.go:122] "RemoveContainer" containerID="edb2364ded11cde72c7274ab93cb50b488e48ac7190401e56fa01a9a527cb493"
Oct 30 12:05:58.918104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.918079    5060 scope.go:122] "RemoveContainer" containerID="8314b993fd06ccfc59ae8a6ef226d00193511b021cbc07320aa9451473591d83"
Oct 30 12:05:58.929651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.929626    5060 scope.go:122] "RemoveContainer" containerID="b50b8f0169df6f61500981df78b619d5ffcce81c3f1e46c92dc31a11e6df9506"
Oct 30 12:05:58.931270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.931244    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.931408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.931326    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdk9\" (UniqueName: \"kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.931408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.931373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.931947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.931927    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.933995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.933975    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:58.955938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:58.955874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdk9\" (UniqueName: \"kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9\") pod \"maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:59.028052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:59.028016    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:05:59.334085 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:05:59.334043    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee348e0_2303_473d_a96b_6ff6ae4809d9.slice/crio-18601abdd66c46dcc20ac0b519ced6a00931eb9e77c7913106e77e0146208b5b WatchSource:0}: Error finding container 18601abdd66c46dcc20ac0b519ced6a00931eb9e77c7913106e77e0146208b5b: Status 404 returned error can't find the container with id 18601abdd66c46dcc20ac0b519ced6a00931eb9e77c7913106e77e0146208b5b
Oct 30 12:05:59.337188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:59.337162    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"]
Oct 30 12:05:59.457236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:05:59.457207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:00.129968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:00.129931    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb" event={"ID":"cee348e0-2303-473d-a96b-6ff6ae4809d9","Type":"ContainerStarted","Data":"d456c8190d71de37bc0e5be5a47bea9b506e2a8c17e25bbbf844912101d5c312"}
Oct 30 12:06:00.130430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:00.129996    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb" event={"ID":"cee348e0-2303-473d-a96b-6ff6ae4809d9","Type":"ContainerStarted","Data":"18601abdd66c46dcc20ac0b519ced6a00931eb9e77c7913106e77e0146208b5b"}
Oct 30 12:06:00.175768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:00.175714    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb" podStartSLOduration=2.175697241 podStartE2EDuration="2.175697241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:00.171082046 +0000 UTC m=+2163.965966540" watchObservedRunningTime="2026-10-30 12:06:00.175697241 +0000 UTC m=+2163.970581732"
Oct 30 12:06:01.140201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:01.140161    5060 generic.go:356] "Generic (PLEG): container finished" podID="cee348e0-2303-473d-a96b-6ff6ae4809d9" containerID="d456c8190d71de37bc0e5be5a47bea9b506e2a8c17e25bbbf844912101d5c312" exitCode=0
Oct 30 12:06:01.140755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:01.140244    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb" event={"ID":"cee348e0-2303-473d-a96b-6ff6ae4809d9","Type":"ContainerDied","Data":"d456c8190d71de37bc0e5be5a47bea9b506e2a8c17e25bbbf844912101d5c312"}
Oct 30 12:06:01.406813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:01.406719    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:02.325497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.325472    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:06:02.476343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.476298    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9\" (UniqueName: \"kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9\") pod \"cee348e0-2303-473d-a96b-6ff6ae4809d9\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") "
Oct 30 12:06:02.476587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.476541    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca\") pod \"cee348e0-2303-473d-a96b-6ff6ae4809d9\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") "
Oct 30 12:06:02.476769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.476751    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca\") pod \"cee348e0-2303-473d-a96b-6ff6ae4809d9\" (UID: \"cee348e0-2303-473d-a96b-6ff6ae4809d9\") "
Oct 30 12:06:02.477601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.477565    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca" pod "cee348e0-2303-473d-a96b-6ff6ae4809d9" (UID: "cee348e0-2303-473d-a96b-6ff6ae4809d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:06:02.479185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.479156    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9" pod "cee348e0-2303-473d-a96b-6ff6ae4809d9" (UID: "cee348e0-2303-473d-a96b-6ff6ae4809d9"). InnerVolumeSpecName "kube-api-access-xmdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:06:02.479749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.479722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca" pod "cee348e0-2303-473d-a96b-6ff6ae4809d9" (UID: "cee348e0-2303-473d-a96b-6ff6ae4809d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:06:02.579001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.578959    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:02.579001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.578992    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee348e0-2303-473d-a96b-6ff6ae4809d9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:02.579001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:02.579001    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmdk9\" (UniqueName: \"kubernetes.io/projected/cee348e0-2303-473d-a96b-6ff6ae4809d9-kube-api-access-xmdk9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:03.160262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:03.160238    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"
Oct 30 12:06:03.160466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:03.160283    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb" event={"ID":"cee348e0-2303-473d-a96b-6ff6ae4809d9","Type":"ContainerDied","Data":"18601abdd66c46dcc20ac0b519ced6a00931eb9e77c7913106e77e0146208b5b"}
Oct 30 12:06:03.160466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:03.160341    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18601abdd66c46dcc20ac0b519ced6a00931eb9e77c7913106e77e0146208b5b"
Oct 30 12:06:03.261742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:03.260222    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:05.035697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:05.035665    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"]
Oct 30 12:06:05.047950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:05.047915    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076723b0-lvx5w"]
Oct 30 12:06:05.101336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:05.101299    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:06.338413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:06.338381    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef4f339-d4a4-45cb-9279-fb01c4ae0599" path="/var/lib/kubelet/pods/bef4f339-d4a4-45cb-9279-fb01c4ae0599/volumes"
Oct 30 12:06:06.913413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:06.913374    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:07.183791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:07.183708    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:07.782152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:07.782113    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:08.252832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.252795    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"]
Oct 30 12:06:08.254023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.253995    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cee348e0-2303-473d-a96b-6ff6ae4809d9" containerName="revoke-keys"
Oct 30 12:06:08.254195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.254181    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee348e0-2303-473d-a96b-6ff6ae4809d9" containerName="revoke-keys"
Oct 30 12:06:08.254555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.254540    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee348e0-2303-473d-a96b-6ff6ae4809d9" containerName="revoke-keys"
Oct 30 12:06:08.262141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.262116    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275029    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz4h\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275128    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275198    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275272    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275330    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275375    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.276276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.275492    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.296242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.296202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.303318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.303248    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"]
Oct 30 12:06:08.333015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.332966    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bbae89-openshift-default-dockercfg-j9gsf\""
Oct 30 12:06:08.341648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.341613    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:08.408621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.408539    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.409368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.409348    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.411147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.411084    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.412078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.412059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz4h\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.412670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.412651    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.413137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.413119    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.413711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.413693    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.414154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.414138    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.414635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.414092    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.414770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.409209    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.414863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.411983    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.414952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.414535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.415116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.415100    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.415549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.415428    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.420758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.420730    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.423206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.421027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.439887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.439850    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.440348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.440325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz4h\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h\") pod \"e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") " pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.614169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.613730    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:08.837153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:08.837113    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:09.038299 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:06:09.038262    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96682ef0_5f44_4ee2_96be_59d56a7099a4.slice/crio-7c30722ec7930095201d6bb2cb020c19854a5ccc462d4174f0237743f0312664 WatchSource:0}: Error finding container 7c30722ec7930095201d6bb2cb020c19854a5ccc462d4174f0237743f0312664: Status 404 returned error can't find the container with id 7c30722ec7930095201d6bb2cb020c19854a5ccc462d4174f0237743f0312664
Oct 30 12:06:09.039882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.039856    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"]
Oct 30 12:06:09.041860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.041822    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:09.041978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.041906    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:09.041978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.041945    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:09.087723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.085476    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:09.417297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.415951    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" event={"ID":"96682ef0-5f44-4ee2-96be-59d56a7099a4","Type":"ContainerStarted","Data":"33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"}
Oct 30 12:06:09.417297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.416018    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" event={"ID":"96682ef0-5f44-4ee2-96be-59d56a7099a4","Type":"ContainerStarted","Data":"7c30722ec7930095201d6bb2cb020c19854a5ccc462d4174f0237743f0312664"}
Oct 30 12:06:09.628478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.622773    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:09.650665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.650633    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:09.814509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.814426    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podStartSLOduration=1.81440354 podStartE2EDuration="1.81440354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:09.649402891 +0000 UTC m=+2173.444287423" watchObservedRunningTime="2026-10-30 12:06:09.81440354 +0000 UTC m=+2173.609288035"
Oct 30 12:06:09.830054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:09.830016    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:10.374481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.368494    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:10.396481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.392646    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 12:06:10.616065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.616020    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:06:10.616356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.616323    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:06:10.656658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.656546    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:10.806941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.806556    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:10.816400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:10.815584    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:11.310993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.310947    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:11.338841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.338792    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:11.370532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.370489    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:11.371331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.371309    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 12:06:11.394607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.394567    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:11.395729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.395705    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 12:06:11.400082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.400053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:11.535780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.534128    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 12:06:11.756796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.747881    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 12:06:11.756796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.753137    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:06:11.801151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.801109    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 12:06:11.802544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.802523    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:11.850327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.850283    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 12:06:11.851359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:11.851334    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 12:06:12.043535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:12.042037    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 12:06:12.772544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:12.770765    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:12.887476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:12.886938    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:12.887476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:12.887054    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:06:13.451484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:13.447594    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:13.512481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:13.511568    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:13.732559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:13.731089    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 12:06:15.162727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:15.162688    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:15.940480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:15.939952    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:16.071502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.071409    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"]
Oct 30 12:06:16.126204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.126118    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:16.138522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.128286    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c3ad80-mgvqb"]
Oct 30 12:06:16.288075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.288035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:16.359183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.359146    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee348e0-2303-473d-a96b-6ff6ae4809d9" path="/var/lib/kubelet/pods/cee348e0-2303-473d-a96b-6ff6ae4809d9/volumes"
Oct 30 12:06:16.454055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.453998    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:16.610565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.610405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:16.836630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:16.836528    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:17.336864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:17.336689    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:17.824595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:17.824493    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:18.302433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:18.302353    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:19.696698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:19.696655    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:06:19.697360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:19.697293    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:19.869199 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:06:19.869122    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:06:20.044446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.044210    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:20.323118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.323024    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:20.594928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.594838    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"]
Oct 30 12:06:20.626702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.626663    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"]
Oct 30 12:06:20.626902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.626775    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759232    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759302    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759503    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759599    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759671    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759710    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759767    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.760060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.759810    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9s9\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861141    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861200    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861261    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861340    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861389    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861429    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9s9\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861517    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861551    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861790    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.861991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.861909    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.862122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.862057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.862484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.862440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.862623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.862539    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.864963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.864908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.865089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.864984    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.887749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.887680    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.890020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.889955    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9s9\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:20.975928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:20.975893    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:21.318052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:21.318014    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:21.367553 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:06:21.367520    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c93c71_1617_4599_8f89_e30501be84b4.slice/crio-922e477753e18373a1e1209695a0741084d5e0ec825c8474ca53cf366246013c WatchSource:0}: Error finding container 922e477753e18373a1e1209695a0741084d5e0ec825c8474ca53cf366246013c: Status 404 returned error can't find the container with id 922e477753e18373a1e1209695a0741084d5e0ec825c8474ca53cf366246013c
Oct 30 12:06:21.370294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:21.370261    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:21.370389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:21.370333    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:21.370389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:21.370366    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:21.371610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:21.371584    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"]
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:21.503401    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.165234    5060 generic.go:356] "Generic (PLEG): container finished" podID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerID="d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8" exitCode=0
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.165275    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" event={"ID":"9a37c5b7-b5a1-4389-baa2-692562ae1b96","Type":"ContainerDied","Data":"d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"}
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.165938    5060 scope.go:122] "RemoveContainer" containerID="d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.167588    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" event={"ID":"76c93c71-1617-4599-8f89-e30501be84b4","Type":"ContainerStarted","Data":"0ef3e57a119d5b3192a16866f58fac2eeed110de8249dafef8c84290997b59bd"}
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.167637    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" event={"ID":"76c93c71-1617-4599-8f89-e30501be84b4","Type":"ContainerStarted","Data":"922e477753e18373a1e1209695a0741084d5e0ec825c8474ca53cf366246013c"}
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.167761    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.167823    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.167857    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.339324    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.355850    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podStartSLOduration=2.355827386 podStartE2EDuration="2.355827386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:22.32788363 +0000 UTC m=+2186.122768125" watchObservedRunningTime="2026-10-30 12:06:22.355827386 +0000 UTC m=+2186.150711883"
Oct 30 12:06:22.700607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.509515    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:23.058809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.978185    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:23.058809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.982009    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:23.058809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:22.982065    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:23.195762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.193428    5060 generic.go:356] "Generic (PLEG): container finished" podID="7a395608-0ff6-42de-adb2-c425bae17902" containerID="d7ed3f075a69da122994c708abda8d8759ecf6cbea893dffb5abff5bd62ef319" exitCode=0
Oct 30 12:06:23.195762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.193551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" event={"ID":"7a395608-0ff6-42de-adb2-c425bae17902","Type":"ContainerDied","Data":"d7ed3f075a69da122994c708abda8d8759ecf6cbea893dffb5abff5bd62ef319"}
Oct 30 12:06:23.195762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.194051    5060 scope.go:122] "RemoveContainer" containerID="d7ed3f075a69da122994c708abda8d8759ecf6cbea893dffb5abff5bd62ef319"
Oct 30 12:06:23.206898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.204906    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:23.206898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.205016    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:23.206898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.205092    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:23.244670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.243076    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" event={"ID":"9a37c5b7-b5a1-4389-baa2-692562ae1b96","Type":"ContainerStarted","Data":"007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53"}
Oct 30 12:06:23.456552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.456519    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:23.589285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.588282    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"]
Oct 30 12:06:23.702972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.697183    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:23.715816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.715784    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"openshift-service-ca.crt\""
Oct 30 12:06:23.727531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.716443    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"default-dockercfg-rxvrs\""
Oct 30 12:06:23.727687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.717122    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"auth-test-model-f1b14f-kserve-self-signed-certs\""
Oct 30 12:06:23.727687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.717198    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"kube-root-ca.crt\""
Oct 30 12:06:23.727687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.726378    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"]
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.891320    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.892202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9sw\" (UniqueName: \"kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.892248    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.892284    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.892342    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.892425    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:23.899489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.892591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.016537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.997714    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:24.016537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:23.997788    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.032380    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.033007    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.033050    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.035092    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.035255    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.035328    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9sw\" (UniqueName: \"kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.035359    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.035393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.035477    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.037480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.037343    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.038214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.037683    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.047211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.039095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.173477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.172525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.173975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.173946    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9sw\" (UniqueName: \"kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.261477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.257401    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm\") pod \"auth-test-model-f1b14f-kserve-5c58677765-jkqbv\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.266485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.266433    5060 generic.go:356] "Generic (PLEG): container finished" podID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerID="d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17" exitCode=0
Oct 30 12:06:24.266730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.266494    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" event={"ID":"cb182d45-f45a-46c8-a9b4-0d2d97113b72","Type":"ContainerDied","Data":"d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"}
Oct 30 12:06:24.267061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.267034    5060 scope.go:122] "RemoveContainer" containerID="d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"
Oct 30 12:06:24.269190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.269157    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:24.269297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.269235    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:24.269297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.269244    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" event={"ID":"7a395608-0ff6-42de-adb2-c425bae17902","Type":"ContainerStarted","Data":"0fc95420432a40ceb3e9bf5c4fcee1567774b6264952ae8eb55bb40252870895"}
Oct 30 12:06:24.269297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.269284    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:24.415125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.413884    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:24.900320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.900235    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:06:24.921780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.910183    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:24.921780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.910252    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:24.928917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.928838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:24.955122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.955087    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:24.955428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.955397    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:24.993477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.987882    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:24.993477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:24.987947    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:25.173784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.169343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:25.393558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.370584    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:25.835857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.834112    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:25.835857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.834183    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:25.850515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.850476    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" event={"ID":"cb182d45-f45a-46c8-a9b4-0d2d97113b72","Type":"ContainerStarted","Data":"5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce"}
Oct 30 12:06:25.915521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.888547    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"]
Oct 30 12:06:25.915521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.889665    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" containerID="cri-o://23910df04bdbce9f827e7543666c9e541720f7c6467a40a97b1e094257605f50" gracePeriod=30
Oct 30 12:06:25.951760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.951727    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:25.952764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.952728    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:25.988499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.987601    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:25.988499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:25.987671    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:26.029099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.029061    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"]
Oct 30 12:06:26.294472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.294357    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:06:26.318046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.318008    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:26.337847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.337793    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:26.592121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.592030    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:06:26.592493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.592468    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:26.622706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.622663    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:26.685746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.619972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:26.761720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.728619    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:06:26.781480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.777591    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 12:06:26.783923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.783664    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:26.842730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.833557    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:26.842730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.835333    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:26.842730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.835380    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:26.919685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.919645    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" event={"ID":"6024bc15-b94a-4f59-8342-9fc6109b8bd1","Type":"ContainerStarted","Data":"c709806636d83a91691de20a39425f27e2f0ca3396805f781c88bc46f921add8"}
Oct 30 12:06:26.942781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.942747    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:26.963728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.963685    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:26.964049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.963999    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:26.987315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.987278    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:26.987620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:26.987593    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:27.076095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.076054    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:27.338658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.338612    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:27.362488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.340093    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 12:06:27.363656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.355520    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 12:06:27.365302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.363897    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:27.432120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.425666    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:27.452898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.452856    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 12:06:27.594115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.577829    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:27.594115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.577891    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:27.769732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.769685    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:27.811530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.783929    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:27.811530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.796726    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 12:06:27.811530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.796841    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 12:06:27.854274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.850524    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:27.854274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.850576    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:27.973385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.973354    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:27.974027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:27.973970    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:28.021246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.021162    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:28.021561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.021511    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:28.077908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.077863    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 12:06:28.086512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.081875    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 12:06:28.352518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.339531    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:06:28.355105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.355050    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:28.364598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.364545    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:28.426102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.412422    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:28.499525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.442423    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:06:28.616115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.616019    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:28.616384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.616352    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:28.782476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.761161    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:28.782793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.780018    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:06:28.784140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.784110    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 12:06:28.969543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.934647    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:28.969543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.934704    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:28.970315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.970279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:28.970814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.970783    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:28.971022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.970987    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:28.993781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.993745    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 12:06:28.994607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.994376    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 12:06:28.995904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.995878    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 12:06:28.996409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:28.996392    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 12:06:29.112498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.080683    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 12:06:29.112498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.087685    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.367545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.363839    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 12:06:29.367545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.364387    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 12:06:29.412145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.412091    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.436752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.436722    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:29.518218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.518157    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:06:29.729716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.729675    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.823481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.797759    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:29.823481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.797830    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:29.823481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.799827    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.823481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.803920    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.823481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.804660    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:06:29.854479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.852839    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.858484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.858422    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.881487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.878512    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:29.881487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.878592    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:29.907935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.907889    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:29.971475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.965042    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:29.971871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.971821    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:30.008536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.990065    5060 patch_prober.go:29] interesting pod/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:06:30.008536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:29.990136    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:06:30.084612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.084555    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:30.230927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.230861    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" podStartSLOduration=7.230840447 podStartE2EDuration="7.230840447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:29.796550736 +0000 UTC m=+2193.591435231" watchObservedRunningTime="2026-10-30 12:06:30.230840447 +0000 UTC m=+2194.025724944"
Oct 30 12:06:30.231699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.231671    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:30.231823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.231754    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"]
Oct 30 12:06:30.466098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.463595    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"]
Oct 30 12:06:30.489939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.489902    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:30.509193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.509138    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:30.520671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.520618    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:30.658976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.658932    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:06:30.659444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.659427    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:06:30.660737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.659779    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:06:30.660897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.659909    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:06:30.699992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.697624    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:30.699992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.697705    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:30.727624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.720012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:30.784008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.767922    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 12:06:30.948303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.812906    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 12:06:30.989596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.813097    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:06:30.990211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.813127    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 12:06:30.997577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.814121    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:31.024481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.814725    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.028276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.028246    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.038143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.038110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsl5\" (UniqueName: \"kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.929587    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.948661    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" event={"ID":"6024bc15-b94a-4f59-8342-9fc6109b8bd1","Type":"ContainerStarted","Data":"f078536878a7ceee17360d1e2c2f204f909e24d396c43ee95e3e96131dbb208a"}
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.076713    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.076800    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.862549    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.076825    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.076878    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.871233    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.077073    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:30.990917    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 12:06:31.084619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.077123    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 12:06:31.116864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.116816    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:31.195837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.178144    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsl5\" (UniqueName: \"kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.195837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.178412    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.195837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.178485    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.207523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.205333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.449824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.413558    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 12:06:31.464396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.464271    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 12:06:31.465818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.465797    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 12:06:31.533697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.533651    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:06:31.654409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.642406    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsl5\" (UniqueName: \"kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.695495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.677761    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bf6585a-c977j\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.739270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.696988    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 12:06:31.739270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.699392    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 12:06:31.739270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.714715    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:31.739270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.714793    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:31.767180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.767149    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 12:06:31.793377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.793344    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:31.819694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.800572    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 12:06:31.835545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.835511    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 12:06:31.835822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.835791    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 12:06:31.838419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.838394    5060 generic.go:356] "Generic (PLEG): container finished" podID="5fc01688-2286-4f28-829d-ca31b876f398" containerID="23910df04bdbce9f827e7543666c9e541720f7c6467a40a97b1e094257605f50" exitCode=0
Oct 30 12:06:31.863479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.863176    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" event={"ID":"5fc01688-2286-4f28-829d-ca31b876f398","Type":"ContainerDied","Data":"23910df04bdbce9f827e7543666c9e541720f7c6467a40a97b1e094257605f50"}
Oct 30 12:06:31.863479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.863328    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 12:06:31.863817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.863776    5060 scope.go:122] "RemoveContainer" containerID="9446f42d1486d34dc2ab98b2dea902297f132a1d17efd5ab534d59bffcfc0c0f"
Oct 30 12:06:31.976314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.976051    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:06:31.984777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:31.980215    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:06:32.065648 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:06:32.063401    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc01688_2286_4f28_829d_ca31b876f398.slice/crio-conmon-23910df04bdbce9f827e7543666c9e541720f7c6467a40a97b1e094257605f50.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 12:06:32.564152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.564024    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:32.570580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.570557    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"]
Oct 30 12:06:32.654967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.575591    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 12:06:32.654967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.575658    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 12:06:32.667032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.666999    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:06:32.750732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.750704    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.750992    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751185    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751280    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751521    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751760    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.751911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751898    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.752083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.752002    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo\") pod \"5fc01688-2286-4f28-829d-ca31b876f398\" (UID: \"5fc01688-2286-4f28-829d-ca31b876f398\") "
Oct 30 12:06:32.752587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751684    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:06:32.752711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751860    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:06:32.752781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.751991    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:06:32.752847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.752104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:06:32.752903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.752282    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:06:32.753047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.753022    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.753095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.753054    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.753095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.753068    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.753095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.753082    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.754164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.754127    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "kube-api-access-zv5gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:06:32.754908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.754839    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:06:32.754908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.754873    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:06:32.755654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.755626    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo" pod "5fc01688-2286-4f28-829d-ca31b876f398" (UID: "5fc01688-2286-4f28-829d-ca31b876f398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:06:32.839160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.839085    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:06:32.840285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.840267    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:06:32.854073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.854043    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.854073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.854072    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc01688-2286-4f28-829d-ca31b876f398-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.854310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.854084    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc01688-2286-4f28-829d-ca31b876f398-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.854310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.854096    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc01688-2286-4f28-829d-ca31b876f398-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.854310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.854109    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5gf\" (UniqueName: \"kubernetes.io/projected/5fc01688-2286-4f28-829d-ca31b876f398-kube-api-access-zv5gf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:32.855935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.855902    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j" event={"ID":"78b3a431-f821-41e4-a4bd-32b463be5b23","Type":"ContainerStarted","Data":"9b18046b83ad330f2ca2edb9657a382c02717c544667a3884cd7768efd4b5b83"}
Oct 30 12:06:32.857344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.857314    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"
Oct 30 12:06:32.857468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.857351    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww" event={"ID":"5fc01688-2286-4f28-829d-ca31b876f398","Type":"ContainerDied","Data":"3d17f2ba5649466791eb443f23428f7d78bf91bc2676c4905c44fbae11e2a78d"}
Oct 30 12:06:32.857468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:32.857415    5060 scope.go:122] "RemoveContainer" containerID="23910df04bdbce9f827e7543666c9e541720f7c6467a40a97b1e094257605f50"
Oct 30 12:06:33.025097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.025024    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"]
Oct 30 12:06:33.031530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.031435    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c3ad80-openshift-default-7fc847b985-hk4ww"]
Oct 30 12:06:33.574430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.574396    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:06:33.574903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.574792    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:06:33.575547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.575528    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:06:33.867953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.867860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j" event={"ID":"78b3a431-f821-41e4-a4bd-32b463be5b23","Type":"ContainerStarted","Data":"6787fc22528b21632c428f994100a8ef88f16d4272d85921d956e1760d7161ce"}
Oct 30 12:06:33.911572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:33.911508    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j" podStartSLOduration=4.91148823 podStartE2EDuration="4.91148823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:33.910952217 +0000 UTC m=+2197.705836725" watchObservedRunningTime="2026-10-30 12:06:33.91148823 +0000 UTC m=+2197.706372727"
Oct 30 12:06:34.158250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.158169    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:34.336612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.336580    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc01688-2286-4f28-829d-ca31b876f398" path="/var/lib/kubelet/pods/5fc01688-2286-4f28-829d-ca31b876f398/volumes"
Oct 30 12:06:34.416833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.416752    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:34.432640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.432607    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:34.890246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.890208    5060 generic.go:356] "Generic (PLEG): container finished" podID="78b3a431-f821-41e4-a4bd-32b463be5b23" containerID="6787fc22528b21632c428f994100a8ef88f16d4272d85921d956e1760d7161ce" exitCode=0
Oct 30 12:06:34.890696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.890290    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j" event={"ID":"78b3a431-f821-41e4-a4bd-32b463be5b23","Type":"ContainerDied","Data":"6787fc22528b21632c428f994100a8ef88f16d4272d85921d956e1760d7161ce"}
Oct 30 12:06:34.907674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:34.907645    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:06:35.052747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:35.052709    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"]
Oct 30 12:06:35.057447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:35.057419    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d980dcdd-rbjm7"]
Oct 30 12:06:36.068901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.068867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:36.076117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.076082    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:36.233505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.233339    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca\") pod \"78b3a431-f821-41e4-a4bd-32b463be5b23\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") "
Oct 30 12:06:36.233505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.233435    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca\") pod \"78b3a431-f821-41e4-a4bd-32b463be5b23\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") "
Oct 30 12:06:36.236309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.234030    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5\" (UniqueName: \"kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5\") pod \"78b3a431-f821-41e4-a4bd-32b463be5b23\" (UID: \"78b3a431-f821-41e4-a4bd-32b463be5b23\") "
Oct 30 12:06:36.236309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.235810    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca" pod "78b3a431-f821-41e4-a4bd-32b463be5b23" (UID: "78b3a431-f821-41e4-a4bd-32b463be5b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:06:36.251879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.251756    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca" pod "78b3a431-f821-41e4-a4bd-32b463be5b23" (UID: "78b3a431-f821-41e4-a4bd-32b463be5b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:06:36.252259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.252205    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5" pod "78b3a431-f821-41e4-a4bd-32b463be5b23" (UID: "78b3a431-f821-41e4-a4bd-32b463be5b23"). InnerVolumeSpecName "kube-api-access-9bsl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:06:36.335405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.335376    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:36.335405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.335407    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsl5\" (UniqueName: \"kubernetes.io/projected/78b3a431-f821-41e4-a4bd-32b463be5b23-kube-api-access-9bsl5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:36.335652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.335422    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b3a431-f821-41e4-a4bd-32b463be5b23-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:06:36.336656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.336635    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6c2540-ad15-4451-80a7-a6ef26108a77" path="/var/lib/kubelet/pods/ed6c2540-ad15-4451-80a7-a6ef26108a77/volumes"
Oct 30 12:06:36.916551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.916523    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"
Oct 30 12:06:36.916551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.916528    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j" event={"ID":"78b3a431-f821-41e4-a4bd-32b463be5b23","Type":"ContainerDied","Data":"9b18046b83ad330f2ca2edb9657a382c02717c544667a3884cd7768efd4b5b83"}
Oct 30 12:06:36.916839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:36.916594    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b18046b83ad330f2ca2edb9657a382c02717c544667a3884cd7768efd4b5b83"
Oct 30 12:06:37.872680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:37.872597    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:38.036613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:38.036561    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"]
Oct 30 12:06:38.045210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:38.045175    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb0a08-pjb5f"]
Oct 30 12:06:38.335421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:38.335389    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb955f2-055e-4bd3-bf2f-de8027cf7892" path="/var/lib/kubelet/pods/acb955f2-055e-4bd3-bf2f-de8027cf7892/volumes"
Oct 30 12:06:39.816611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:39.816566    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:41.017976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.017921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:41.412320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.412224    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"]
Oct 30 12:06:41.413022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.412999    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy"
Oct 30 12:06:41.413153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413023    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy"
Oct 30 12:06:41.413153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413037    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy"
Oct 30 12:06:41.413153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413045    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy"
Oct 30 12:06:41.413153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413100    5060 cpu_manager.go:414] "containerMap: removing container" podUID="78b3a431-f821-41e4-a4bd-32b463be5b23" containerName="revoke-keys"
Oct 30 12:06:41.413153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413109    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b3a431-f821-41e4-a4bd-32b463be5b23" containerName="revoke-keys"
Oct 30 12:06:41.413423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413298    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy"
Oct 30 12:06:41.413423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413311    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc01688-2286-4f28-829d-ca31b876f398" containerName="istio-proxy"
Oct 30 12:06:41.413423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.413328    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b3a431-f821-41e4-a4bd-32b463be5b23" containerName="revoke-keys"
Oct 30 12:06:41.418569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.418544    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.448369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.448334    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"]
Oct 30 12:06:41.454589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.454553    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a51616-openshift-default-dockercfg-gcsq8\""
Oct 30 12:06:41.518209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.518176    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:41.527850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.527817    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.527850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.527852    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66vh2\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.527987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.528036    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.528198    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.528234    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.528259    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.528311    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.528528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.528515    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630400    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630485    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630512    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630596    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630739    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630822    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630850    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66vh2\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.630964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630894    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.631325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630932    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.631325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.631525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.631368    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.631525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.631416    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.631525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.630999    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.631861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.631181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.633592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.633564    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.633987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.633965    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.655141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.655113    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.655346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.655322    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66vh2\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2\") pod \"e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") " pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.676915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.676816    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:41.745800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.745763    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:41.863477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:41.844079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:42.261331 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:06:42.261303    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ec294a_c5b8_4c9f_b7ab_f6babf112b99.slice/crio-b86c83e25db12d2c98a66c9f1af4a02dc5f22af3d50aaac71e425f398a04e1e7 WatchSource:0}: Error finding container b86c83e25db12d2c98a66c9f1af4a02dc5f22af3d50aaac71e425f398a04e1e7: Status 404 returned error can't find the container with id b86c83e25db12d2c98a66c9f1af4a02dc5f22af3d50aaac71e425f398a04e1e7
Oct 30 12:06:42.264776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:42.264751    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"]
Oct 30 12:06:42.264961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:42.264915    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:42.265068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:42.265003    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:42.265068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:42.265046    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:06:42.983469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:42.983417    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" event={"ID":"10ec294a-c5b8-4c9f-b7ab-f6babf112b99","Type":"ContainerStarted","Data":"456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"}
Oct 30 12:06:42.983676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:42.983498    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" event={"ID":"10ec294a-c5b8-4c9f-b7ab-f6babf112b99","Type":"ContainerStarted","Data":"b86c83e25db12d2c98a66c9f1af4a02dc5f22af3d50aaac71e425f398a04e1e7"}
Oct 30 12:06:43.042148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.042083    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podStartSLOduration=2.042062795 podStartE2EDuration="2.042062795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:43.040742015 +0000 UTC m=+2206.835626509" watchObservedRunningTime="2026-10-30 12:06:43.042062795 +0000 UTC m=+2206.836947290"
Oct 30 12:06:43.615081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.615037    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:43.748159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.748052    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:43.772126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.772091    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:43.783522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.782116    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:43.993972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.993936    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:43.994849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:43.994819    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:06:45.602700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:45.602664    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:47.465785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:47.465736    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:47.647797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:47.647545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:47.805674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:47.805582    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:48.052650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:48.052615    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"]
Oct 30 12:06:48.061201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:48.061121    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd2e292-cjdxw"]
Oct 30 12:06:48.364047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:48.363941    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d190f090-23ff-46be-85f0-c001db92d9e7" path="/var/lib/kubelet/pods/d190f090-23ff-46be-85f0-c001db92d9e7/volumes"
Oct 30 12:06:49.635098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:49.635060    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:49.859801 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:06:49.859768    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:06:50.034598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:50.034555    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"]
Oct 30 12:06:50.046161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:50.046121    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ceb824e-jnhzm"]
Oct 30 12:06:50.336200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:50.336114    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f86870-543e-4547-a11b-8c2d401f75b6" path="/var/lib/kubelet/pods/35f86870-543e-4547-a11b-8c2d401f75b6/volumes"
Oct 30 12:06:51.253698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:51.253659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:51.465511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:51.465474    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:53.250635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:53.250593    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:55.167994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.167955    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:55.485654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.485611    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:55.613675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.613640    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"]
Oct 30 12:06:55.619329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.619304    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.645762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.645728    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"]
Oct 30 12:06:55.651428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.651406    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"auth-test-model-4fff4f-kserve-self-signed-certs\""
Oct 30 12:06:55.651643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.651626    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"kube-root-ca.crt\""
Oct 30 12:06:55.651702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.651679    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"openshift-service-ca.crt\""
Oct 30 12:06:55.654222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.654205    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"default-dockercfg-l9bt7\""
Oct 30 12:06:55.787346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.786567    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv2h\" (UniqueName: \"kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.787346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.786790    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.787346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.786858    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.787346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.787029    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.787346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.787053    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.787346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.787110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.836814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.836773    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:55.889098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889305    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889410    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv2h\" (UniqueName: \"kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889541    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889566    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889629    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.889827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.889806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.890055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.890038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.894605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.894575    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.897916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.897887    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.927575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.927536    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv2h\" (UniqueName: \"kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h\") pod \"auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:55.941501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:55.939812    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:56.002541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:56.000905    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:56.381298 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:06:56.381266    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215f0aaf_a182_4548_ad6d_aac08bde0a60.slice/crio-d31ee9ce8124de1feb449146e8d9c0678f301d7bd742c9318bc61e94c36614c0 WatchSource:0}: Error finding container d31ee9ce8124de1feb449146e8d9c0678f301d7bd742c9318bc61e94c36614c0: Status 404 returned error can't find the container with id d31ee9ce8124de1feb449146e8d9c0678f301d7bd742c9318bc61e94c36614c0
Oct 30 12:06:56.391961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:56.391928    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"]
Oct 30 12:06:57.151386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.151344    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg" event={"ID":"215f0aaf-a182-4548-ad6d-aac08bde0a60","Type":"ContainerStarted","Data":"185d197fe3958cf8451978df60a71604b8b1c547a3f251cd3d124239901dbaf0"}
Oct 30 12:06:57.151386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.151390    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg" event={"ID":"215f0aaf-a182-4548-ad6d-aac08bde0a60","Type":"ContainerStarted","Data":"d31ee9ce8124de1feb449146e8d9c0678f301d7bd742c9318bc61e94c36614c0"}
Oct 30 12:06:57.200269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.200213    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg" podStartSLOduration=2.200193277 podStartE2EDuration="2.200193277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:57.194375743 +0000 UTC m=+2220.989260248" watchObservedRunningTime="2026-10-30 12:06:57.200193277 +0000 UTC m=+2220.995077808"
Oct 30 12:06:57.485156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.479471    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:57.775305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.775208    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:57.940670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.940631    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:06:57.950668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:57.950631    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:06:58.996878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:58.996841    5060 scope.go:122] "RemoveContainer" containerID="a93d6d1573842751bf8b12cf20f04c624022977e152af44ef2eec93e11464615"
Oct 30 12:06:59.010305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:59.010276    5060 scope.go:122] "RemoveContainer" containerID="f57baf9d76f2698dce8df95f47bf790141875eef6d9a42dd153069a316d4e3bc"
Oct 30 12:06:59.291525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:06:59.291425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:01.123425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:01.123376    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:02.153587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:02.153545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:03.507253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:03.507218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:05.297487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:05.297425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:05.940427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:05.940391    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:07:05.953117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:05.953086    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:07:06.357174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:06.357145    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:07:07.191475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:07.189437    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:08.903379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:08.903310    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:10.609231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:10.609180    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:12.524595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:12.524489    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:12.946542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:12.944508    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": read tcp 10.132.0.2:33594->10.132.0.236:15021: read: connection reset by peer" start-of-body=
Oct 30 12:07:12.946542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:12.944595    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": read tcp 10.132.0.2:33594->10.132.0.236:15021: read: connection reset by peer"
Oct 30 12:07:13.421680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:13.421646    5060 generic.go:356] "Generic (PLEG): container finished" podID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerID="33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d" exitCode=0
Oct 30 12:07:13.421903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:13.421718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" event={"ID":"96682ef0-5f44-4ee2-96be-59d56a7099a4","Type":"ContainerDied","Data":"33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"}
Oct 30 12:07:13.422297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:13.422276    5060 scope.go:122] "RemoveContainer" containerID="33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"
Oct 30 12:07:13.424342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:13.424310    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:13.424443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:13.424416    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:13.424516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:13.424483    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:14.433976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:14.433942    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" event={"ID":"96682ef0-5f44-4ee2-96be-59d56a7099a4","Type":"ContainerStarted","Data":"9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442"}
Oct 30 12:07:14.535302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:14.535264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:14.618717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:14.618673    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:07:14.618930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:14.618887    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:07:14.619007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:14.618966    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:07:15.614805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:15.614760    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:07:15.615351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:15.614826    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:07:16.616478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:16.615839    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:07:16.616478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:16.615905    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:07:16.735668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:16.735614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:17.402026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:17.401989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:17.614992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:17.614955    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:07:17.615281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:17.615248    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:07:18.163661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:18.163615    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:18.626900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:18.626863    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:07:18.643478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:18.632602    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:07:18.824855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:18.824825    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:19.615528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:19.615437    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 12:07:19.616074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:19.615558    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 12:07:19.858236 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:07:19.858135    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:07:20.594892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.594841    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:20.623689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.623598    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:07:20.624714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.624512    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:07:20.625888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.625866    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:07:20.948954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.948903    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"]
Oct 30 12:07:20.954930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.954890    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:20.969204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.969178    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"]
Oct 30 12:07:20.970353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.970221    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-28ca05\"/\"openshift-service-ca.crt\""
Oct 30 12:07:20.970353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.970257    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-28ca05\"/\"default-dockercfg-89ndd\""
Oct 30 12:07:20.970353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.970329    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-28ca05\"/\"auth-test-model-28ca05-kserve-self-signed-certs\""
Oct 30 12:07:20.970353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:20.970266    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-28ca05\"/\"kube-root-ca.crt\""
Oct 30 12:07:21.026465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.026423    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:21.124104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.124064    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.124349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.124131    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vts7z\" (UniqueName: \"kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.124349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.124272    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.124349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.124325    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.124648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.124620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.124727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.124677    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.229353    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts7z\" (UniqueName: \"kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.229484    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.229547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.229695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.229728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.229829    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.231109    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.231678    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.232530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.231905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.254311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.254281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.264471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.264195    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vts7z\" (UniqueName: \"kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.299153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.299117    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm\") pod \"auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") " pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.572522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.572401    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:21.886675 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:07:21.886639    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5362b1a9_e415_4e66_a911_5fbe5d698490.slice/crio-33b9249cf1933ade75271a2e1b48f039ccb728e3e5cb59de38da4390981b152b WatchSource:0}: Error finding container 33b9249cf1933ade75271a2e1b48f039ccb728e3e5cb59de38da4390981b152b: Status 404 returned error can't find the container with id 33b9249cf1933ade75271a2e1b48f039ccb728e3e5cb59de38da4390981b152b
Oct 30 12:07:21.888600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:21.888561    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"]
Oct 30 12:07:22.035633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.035601    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"]
Oct 30 12:07:22.041043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.041007    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40b7a39-99x2v"]
Oct 30 12:07:22.355619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.351786    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4" path="/var/lib/kubelet/pods/d3759011-dbd0-4ff6-91ac-d7aa12cbc0b4/volumes"
Oct 30 12:07:22.397171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.397134    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:22.536477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.534492    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8" event={"ID":"5362b1a9-e415-4e66-a911-5fbe5d698490","Type":"ContainerStarted","Data":"a5abb8a9930ea7ba0d8e65b744176cacad9530a7c71fe2d6aa41940659ccac9b"}
Oct 30 12:07:22.536477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.534553    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8" event={"ID":"5362b1a9-e415-4e66-a911-5fbe5d698490","Type":"ContainerStarted","Data":"33b9249cf1933ade75271a2e1b48f039ccb728e3e5cb59de38da4390981b152b"}
Oct 30 12:07:22.548003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.547953    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:22.573548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.573508    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:22.587423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.587360    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8" podStartSLOduration=2.587343853 podStartE2EDuration="2.587343853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:07:22.58383675 +0000 UTC m=+2246.378721244" watchObservedRunningTime="2026-10-30 12:07:22.587343853 +0000 UTC m=+2246.382228355"
Oct 30 12:07:22.713721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:22.713560    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:24.191766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:24.191716    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:26.054468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:26.054414    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:27.841321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:27.841288    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:28.983944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:28.983910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:29.664493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:29.664426    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:31.418355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:31.418319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:31.573388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:31.573355    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:31.587159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:31.587127    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:31.632074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:31.632045    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:07:32.035006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:32.034971    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"]
Oct 30 12:07:32.039664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:32.039636    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b662bb8b-msb2z"]
Oct 30 12:07:32.336696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:32.336622    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07da6c7c-0bb6-4668-9ec9-14ca7c181d7c" path="/var/lib/kubelet/pods/07da6c7c-0bb6-4668-9ec9-14ca7c181d7c/volumes"
Oct 30 12:07:32.408914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:32.408881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:32.566719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:32.566683    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:33.144204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:33.144162    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:34.979693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:34.979644    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:35.412886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.408514    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"]
Oct 30 12:07:35.415954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.415926    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.442576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.442544    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4fff4f-openshift-default-dockercfg-bjz7v\""
Oct 30 12:07:35.457913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.457876    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"]
Oct 30 12:07:35.572427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572388    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572447    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572593    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572638    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572682    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdc7\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572725    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572852    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.572898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.572881    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683693    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683739    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683785    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683813    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdc7\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683903    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.683943    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.684029    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.684830    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.702529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.701947    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.703284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.702629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.727985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.703637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.727985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.703915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.755081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.755006    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.791516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.791240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdc7\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.810478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.791821    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.810478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.792040    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:35.853161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:35.851882    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:36.229377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.229337    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"]
Oct 30 12:07:36.229836 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:07:36.229756    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3efebc_c1e2_45ec_88e4_249d399d5194.slice/crio-a2aa7735bf051a41bbf0218537310267fe9993ae5c71684a8a8b00c359c29ecb WatchSource:0}: Error finding container a2aa7735bf051a41bbf0218537310267fe9993ae5c71684a8a8b00c359c29ecb: Status 404 returned error can't find the container with id a2aa7735bf051a41bbf0218537310267fe9993ae5c71684a8a8b00c359c29ecb
Oct 30 12:07:36.232571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.232530    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:36.232693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.232611    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:36.232693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.232641    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:36.845479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.844591    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" event={"ID":"2b3efebc-c1e2-45ec-88e4-249d399d5194","Type":"ContainerStarted","Data":"ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99"}
Oct 30 12:07:36.845479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.844719    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" event={"ID":"2b3efebc-c1e2-45ec-88e4-249d399d5194","Type":"ContainerStarted","Data":"a2aa7735bf051a41bbf0218537310267fe9993ae5c71684a8a8b00c359c29ecb"}
Oct 30 12:07:36.855414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.855377    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:36.917476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:36.917259    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podStartSLOduration=1.917236359 podStartE2EDuration="1.917236359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:07:36.912051733 +0000 UTC m=+2260.706936240" watchObservedRunningTime="2026-10-30 12:07:36.917236359 +0000 UTC m=+2260.712120857"
Oct 30 12:07:37.012674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:37.008984    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:37.862486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:37.861812    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 12:07:37.862486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:37.861891    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 12:07:38.047689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:38.047650    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"]
Oct 30 12:07:38.058024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:38.057986    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e124709-lphr8"]
Oct 30 12:07:38.346188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:38.346154    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6" path="/var/lib/kubelet/pods/b40000dd-ee57-4ec6-9b9b-2b1d7b68f7a6/volumes"
Oct 30 12:07:38.769003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:38.768971    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:38.855929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:38.855893    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 12:07:38.856222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:38.856194    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 12:07:39.856743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:39.856706    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 12:07:39.857294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:39.856790    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 12:07:40.536477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:40.536027    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:40.856991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:40.856896    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 12:07:40.856991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:40.856971    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 12:07:41.855376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:41.855335    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 12:07:41.855679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:41.855399    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 12:07:42.268423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:42.268381    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:42.998475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:42.993660    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:42.998475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:42.995068    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:42.998475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:42.997510    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:07:44.051671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:44.051614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:45.036643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.036600    5060 patch_prober.go:29] interesting pod/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:07:45.036849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.036675    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:07:45.543873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.543838    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"]
Oct 30 12:07:45.550120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.550091    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.567481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.567400    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"]
Oct 30 12:07:45.572064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.572037    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-933195\"/\"kube-root-ca.crt\""
Oct 30 12:07:45.572237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.572149    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-933195\"/\"openshift-service-ca.crt\""
Oct 30 12:07:45.577607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.577581    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-933195\"/\"default-dockercfg-hhs54\""
Oct 30 12:07:45.577757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.577671    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-933195\"/\"auth-test-model-933195-kserve-self-signed-certs\""
Oct 30 12:07:45.618282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.618243    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.618506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.618381    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.618506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.618445    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjdt\" (UniqueName: \"kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.618642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.618610    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.618703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.618679    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.618769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.618719    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.637971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.637933    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:45.719856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.719805    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.720069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.719866    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjdt\" (UniqueName: \"kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.720069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.720011    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.720222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.720083    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.720222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.720123    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.720222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.720187    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.721705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.720819    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.721705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.721103    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.721705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.721357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.725444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.725416    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.727790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.727764    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.746788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.746740    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjdt\" (UniqueName: \"kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt\") pod \"auth-test-model-933195-kserve-86b7bdc6f-7fmrk\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") " pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:45.778295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.778254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:45.882528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:45.877271    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:46.309211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:46.309182    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"]
Oct 30 12:07:47.166327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.166232    5060 generic.go:356] "Generic (PLEG): container finished" podID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerID="456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae" exitCode=0
Oct 30 12:07:47.166327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.166303    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" event={"ID":"10ec294a-c5b8-4c9f-b7ab-f6babf112b99","Type":"ContainerDied","Data":"456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"}
Oct 30 12:07:47.166963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.166878    5060 scope.go:122] "RemoveContainer" containerID="456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"
Oct 30 12:07:47.168544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.168517    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" event={"ID":"4094169e-fd19-437c-b51c-f41fb7ad0432","Type":"ContainerStarted","Data":"49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c"}
Oct 30 12:07:47.168691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.168560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" event={"ID":"4094169e-fd19-437c-b51c-f41fb7ad0432","Type":"ContainerStarted","Data":"03ee0ca5fb84a913b9eee03aa1ad21fe3a2e0bf9de8179b0a05ec4f596a0b4cb"}
Oct 30 12:07:47.168891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.168856    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:47.169021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.168956    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:47.169021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.169015    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:07:47.274520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.274435    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" podStartSLOduration=2.274415873 podStartE2EDuration="2.274415873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:07:47.267817102 +0000 UTC m=+2271.062701593" watchObservedRunningTime="2026-10-30 12:07:47.274415873 +0000 UTC m=+2271.069300366"
Oct 30 12:07:47.525489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.525348    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:47.878026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:47.877940    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:48.191558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:48.191192    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" event={"ID":"10ec294a-c5b8-4c9f-b7ab-f6babf112b99","Type":"ContainerStarted","Data":"efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0"}
Oct 30 12:07:48.746672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:48.746625    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:07:48.746931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:48.746884    5060 patch_prober.go:29] interesting pod/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 12:07:48.747020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:48.746941    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 12:07:49.295467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:49.295402    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:49.746780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:49.746740    5060 patch_prober.go:29] interesting pod/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 12:07:49.746982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:49.746811    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 12:07:49.859827 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:07:49.859777    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:07:50.749487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:50.747866    5060 patch_prober.go:29] interesting pod/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 12:07:50.749487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:50.747930    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 12:07:51.202652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:51.202615    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:51.513198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:51.513101    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:51.746786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:51.746743    5060 patch_prober.go:29] interesting pod/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 12:07:51.746981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:51.746815    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 12:07:52.746794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:52.746760    5060 patch_prober.go:29] interesting pod/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 12:07:52.747268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:52.746823    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 12:07:53.242673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:53.242627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:53.918428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:53.918395    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:07:53.919579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:53.919533    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:07:53.920656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:53.920628    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:07:54.625645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:54.625604    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:55.062161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:55.062116    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:55.877716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:55.877682    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:55.890700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:55.890670    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:56.366173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:56.366144    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:07:56.767291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:56.767250    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:58.552682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:58.552641    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:07:59.057417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.057384    5060 scope.go:122] "RemoveContainer" containerID="afb9e14aa99467fb1be8b39e6e0ac3d370d40ef8f4b9d750bb60ed238e9050d1"
Oct 30 12:07:59.069189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.069169    5060 scope.go:122] "RemoveContainer" containerID="062532ef244b55cc5e7722f8a8143d119a1736e0845b19771362af3f5d975b1c"
Oct 30 12:07:59.081879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.081854    5060 scope.go:122] "RemoveContainer" containerID="c79b5947070a2797d8d182b1d8e5c6170f67482b379b15a6dce6cc9e45d9f85f"
Oct 30 12:07:59.093163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.093133    5060 scope.go:122] "RemoveContainer" containerID="7fdae5b50e85c4091fb4e70f41196ac73afa767def35fc9f825e37ae674d1579"
Oct 30 12:07:59.105359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.105329    5060 scope.go:122] "RemoveContainer" containerID="0c1dcdb05d11ee2805b8b652daf4b0178dd9ba70a954577358913caa410b1a34"
Oct 30 12:07:59.118828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.118801    5060 scope.go:122] "RemoveContainer" containerID="99e40fa5ebe7fb374a0f3b0a601413c50f4c0ef80ef9d1f26c2ef9c60c196d8f"
Oct 30 12:07:59.130366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:07:59.130341    5060 scope.go:122] "RemoveContainer" containerID="73cd1240d98c9acabec8e71847748fa6f251144d6cd08583ebfabca0c2a87e13"
Oct 30 12:08:00.272701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:00.272662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:01.967386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:01.967344    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:03.754642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:03.754603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:05.559263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:05.559227    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:06.764804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:06.764701    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:07.717480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:07.712539    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:09.601644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:09.601607    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:11.032670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:11.032633    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"]
Oct 30 12:08:11.044647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:11.044613    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5071577-bj4ct"]
Oct 30 12:08:11.345082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:11.344989    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:12.336005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:12.335974    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accda864-c967-4160-b47e-a6e66ce05f92" path="/var/lib/kubelet/pods/accda864-c967-4160-b47e-a6e66ce05f92/volumes"
Oct 30 12:08:13.147651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:13.147600    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:14.843021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:14.842974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:16.031383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:16.031344    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"]
Oct 30 12:08:16.044869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:16.044840    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b348edd2-ns9wg"]
Oct 30 12:08:16.337108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:16.337027    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f" path="/var/lib/kubelet/pods/7714013a-ccf5-4ec9-a7db-bfcf38a7bf6f/volumes"
Oct 30 12:08:16.531045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:16.531015    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:18.288315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:18.288282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:19.857638 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:08:19.857600    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:08:20.059364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:20.059327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:20.283985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:20.283948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:21.855887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:21.855849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:23.597034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:23.596994    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:25.304737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:25.304696    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:27.062759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:27.062724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:28.423265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.423210    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"]
Oct 30 12:08:28.430495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.430425    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.446228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.446170    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"]
Oct 30 12:08:28.450563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.450531    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a51616\"/\"default-dockercfg-pg8xz\""
Oct 30 12:08:28.451144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.451122    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a51616\"/\"openshift-service-ca.crt\""
Oct 30 12:08:28.457571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.457549    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a51616\"/\"kube-root-ca.crt\""
Oct 30 12:08:28.457571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.457557    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a51616\"/\"explicit-model-a51616-kserve-self-signed-certs\""
Oct 30 12:08:28.478336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.478106    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.478336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.478249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.478336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.478295    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllhw\" (UniqueName: \"kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.478694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.478432    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.478777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.478751    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.479096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.479070    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.503280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.503247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:28.580924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.580875    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.581155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.580945    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllhw\" (UniqueName: \"kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.581155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.580995    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.581288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.581174    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.581288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.581262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.581422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.581329    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.581841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.581815    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.582088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.582064    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.582273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.582241    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.585280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.585252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.585599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.585506    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.608344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.608280    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllhw\" (UniqueName: \"kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw\") pod \"explicit-model-a51616-kserve-5bd6bd4cc5-r5sks\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") " pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:28.777477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:28.777122    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:29.071469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.071191    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:29.085266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.085203    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"]
Oct 30 12:08:29.100013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.099586    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ca321-5c2lr"]
Oct 30 12:08:29.175338 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:08:29.175298    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2a3118_97a9_4916_b7c3_724cc9d741af.slice/crio-277d114d64f6b8681d3c9988c8000a93536ef362e9a611ea8d61a217b8b47db6 WatchSource:0}: Error finding container 277d114d64f6b8681d3c9988c8000a93536ef362e9a611ea8d61a217b8b47db6: Status 404 returned error can't find the container with id 277d114d64f6b8681d3c9988c8000a93536ef362e9a611ea8d61a217b8b47db6
Oct 30 12:08:29.176871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.176841    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"]
Oct 30 12:08:29.743839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.743797    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" event={"ID":"3a2a3118-97a9-4916-b7c3-724cc9d741af","Type":"ContainerStarted","Data":"7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755"}
Oct 30 12:08:29.744270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.743858    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" event={"ID":"3a2a3118-97a9-4916-b7c3-724cc9d741af","Type":"ContainerStarted","Data":"277d114d64f6b8681d3c9988c8000a93536ef362e9a611ea8d61a217b8b47db6"}
Oct 30 12:08:29.779088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.779052    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:29.790764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:29.790700    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" podStartSLOduration=1.790679746 podStartE2EDuration="1.790679746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:08:29.78531443 +0000 UTC m=+2313.580198925" watchObservedRunningTime="2026-10-30 12:08:29.790679746 +0000 UTC m=+2313.585564241"
Oct 30 12:08:30.336997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:30.336962    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2265fec5-69bb-4b0f-b581-01b8b9d53903" path="/var/lib/kubelet/pods/2265fec5-69bb-4b0f-b581-01b8b9d53903/volumes"
Oct 30 12:08:30.751962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:30.751916    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:32.531658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:32.531616    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:34.345077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:34.345040    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:36.045544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:36.045508    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:37.685272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:37.685224    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:38.778719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:38.778687    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:38.793232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:38.793202    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:38.844872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:38.844847    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:08:39.034979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:39.034900    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"]
Oct 30 12:08:39.042148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:39.042110    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271932b4-2rczb"]
Oct 30 12:08:39.467900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:39.467853    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:40.031202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:40.031161    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"]
Oct 30 12:08:40.043341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:40.043308    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af089860-p6m6j"]
Oct 30 12:08:40.166477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:40.166430    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:40.500246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:40.499023    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc8e275-321c-4ce9-bf96-88481100310b" path="/var/lib/kubelet/pods/3fc8e275-321c-4ce9-bf96-88481100310b/volumes"
Oct 30 12:08:40.502299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:40.502209    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49f799c-806a-48d0-b3b7-5485810bb37f" path="/var/lib/kubelet/pods/d49f799c-806a-48d0-b3b7-5485810bb37f/volumes"
Oct 30 12:08:41.252550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:41.252511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:43.008028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:43.007993    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:44.650621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:44.650572    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:46.337120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:46.337085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:47.030417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:47.030376    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"]
Oct 30 12:08:47.042899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:47.042871    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce9d129-pq2wc"]
Oct 30 12:08:48.075716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:48.075664    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:48.335900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:48.335817    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fdce63-1b85-42fd-a08e-765ff9a33029" path="/var/lib/kubelet/pods/31fdce63-1b85-42fd-a08e-765ff9a33029/volumes"
Oct 30 12:08:49.859652 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:08:49.859618    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:08:50.048950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:50.048910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:51.526344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.526304    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"]
Oct 30 12:08:51.538983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.538948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:51.807357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.807270    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:51.912318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.912278    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"]
Oct 30 12:08:51.916829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.916795    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:51.937914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.937892    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:08:51.940948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.940922    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"]
Oct 30 12:08:51.948750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.948727    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:08:51.948750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.948738    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:08:51.948961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:51.948796    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:08:52.021693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.021517    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:52.037797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.037434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.037797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.037573    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddsnq\" (UniqueName: \"kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.038066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.037880    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.138875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.138789    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.138875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.138840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.139089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.138898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddsnq\" (UniqueName: \"kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.139734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.139703    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.141267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.141247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.163277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.163245    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddsnq\" (UniqueName: \"kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq\") pod \"maas-api-revoke-keys-e2e-explicit-a51616-9zcbf\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.237322 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.237290    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:52.549296 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:08:52.549268    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7346f194_38e3_48cb_bad3_f212fb736ae9.slice/crio-cef13aa19607a00ebabe3f1e3eec13dd24570d313a515ac5a70fc8b94a5d3df6 WatchSource:0}: Error finding container cef13aa19607a00ebabe3f1e3eec13dd24570d313a515ac5a70fc8b94a5d3df6: Status 404 returned error can't find the container with id cef13aa19607a00ebabe3f1e3eec13dd24570d313a515ac5a70fc8b94a5d3df6
Oct 30 12:08:52.550405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.550378    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"]
Oct 30 12:08:52.973472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:52.973427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf" event={"ID":"7346f194-38e3-48cb-bad3-f212fb736ae9","Type":"ContainerStarted","Data":"cef13aa19607a00ebabe3f1e3eec13dd24570d313a515ac5a70fc8b94a5d3df6"}
Oct 30 12:08:53.863053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:53.862964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:53.985608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:53.985570    5060 generic.go:356] "Generic (PLEG): container finished" podID="7346f194-38e3-48cb-bad3-f212fb736ae9" containerID="44050f3f15071a9e6e38dab335a1d65ef12d66408076a41533bfb3dbbc1f41ae" exitCode=0
Oct 30 12:08:53.985812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:53.985657    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf" event={"ID":"7346f194-38e3-48cb-bad3-f212fb736ae9","Type":"ContainerDied","Data":"44050f3f15071a9e6e38dab335a1d65ef12d66408076a41533bfb3dbbc1f41ae"}
Oct 30 12:08:55.179119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.179089    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:55.285732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.285638    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq\" (UniqueName: \"kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq\") pod \"7346f194-38e3-48cb-bad3-f212fb736ae9\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") "
Oct 30 12:08:55.285732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.285721    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca\") pod \"7346f194-38e3-48cb-bad3-f212fb736ae9\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") "
Oct 30 12:08:55.285988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.285781    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca\") pod \"7346f194-38e3-48cb-bad3-f212fb736ae9\" (UID: \"7346f194-38e3-48cb-bad3-f212fb736ae9\") "
Oct 30 12:08:55.286174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.286145    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca" pod "7346f194-38e3-48cb-bad3-f212fb736ae9" (UID: "7346f194-38e3-48cb-bad3-f212fb736ae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:08:55.288487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.288414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq" pod "7346f194-38e3-48cb-bad3-f212fb736ae9" (UID: "7346f194-38e3-48cb-bad3-f212fb736ae9"). InnerVolumeSpecName "kube-api-access-ddsnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:08:55.288607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.288513    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca" pod "7346f194-38e3-48cb-bad3-f212fb736ae9" (UID: "7346f194-38e3-48cb-bad3-f212fb736ae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:08:55.387948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.387916    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:08:55.387948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.387942    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7346f194-38e3-48cb-bad3-f212fb736ae9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:08:55.387948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.387953    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddsnq\" (UniqueName: \"kubernetes.io/projected/7346f194-38e3-48cb-bad3-f212fb736ae9-kube-api-access-ddsnq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:08:55.620986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:55.620893    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:56.008309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:56.008274    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf" event={"ID":"7346f194-38e3-48cb-bad3-f212fb736ae9","Type":"ContainerDied","Data":"cef13aa19607a00ebabe3f1e3eec13dd24570d313a515ac5a70fc8b94a5d3df6"}
Oct 30 12:08:56.008534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:56.008317    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"
Oct 30 12:08:56.008534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:56.008331    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef13aa19607a00ebabe3f1e3eec13dd24570d313a515ac5a70fc8b94a5d3df6"
Oct 30 12:08:56.032003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:56.031973    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"]
Oct 30 12:08:56.043278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:56.043252    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0458b6c7-hp2g7"]
Oct 30 12:08:56.336969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:56.336900    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f767b3a-1fc2-4633-bdfc-feccb7e32cd7" path="/var/lib/kubelet/pods/2f767b3a-1fc2-4633-bdfc-feccb7e32cd7/volumes"
Oct 30 12:08:57.317022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:57.316987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:58.996724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:58.996666    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:08:59.042685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.042643    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"]
Oct 30 12:08:59.049902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.049872    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9996db71-8r228"]
Oct 30 12:08:59.203125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.203088    5060 scope.go:122] "RemoveContainer" containerID="2b9e92b4d1b5e4e4b0d3098c91caa4e11a07f1c818336207b0918680523d51ea"
Oct 30 12:08:59.217197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.217174    5060 scope.go:122] "RemoveContainer" containerID="5c33a682dc017a5932afd5bb2f8a97fefe44a59962ef0bb9608424700ea327da"
Oct 30 12:08:59.228897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.228875    5060 scope.go:122] "RemoveContainer" containerID="91ab0903b903a33eca75a4c736e092a75f40ee389e1968a32bff4aa9a01b7053"
Oct 30 12:08:59.239127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.239108    5060 scope.go:122] "RemoveContainer" containerID="858fc764d1433834776e3db23df2d9547c640cb3738a749da0b0140e5812d8aa"
Oct 30 12:08:59.249201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.249183    5060 scope.go:122] "RemoveContainer" containerID="7572704dcf2d0e0b0374302f56e7350637fa9071553c47e679fcb79ed0973181"
Oct 30 12:08:59.258884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:08:59.258863    5060 scope.go:122] "RemoveContainer" containerID="eb4a6c089e94ef3dd5d72af105c90aa7480489d45e433d37b3c8ec7944fce4de"
Oct 30 12:09:00.340420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:00.340385    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a6f79d-a414-4a17-a47b-958653dfd58d" path="/var/lib/kubelet/pods/f1a6f79d-a414-4a17-a47b-958653dfd58d/volumes"
Oct 30 12:09:00.756010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:00.755972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:02.428484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:02.428419    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:02.576704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:02.576658    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:02.726980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:02.726949    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:04.318899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:04.318845    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:06.084639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.084594    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:06.558648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.558611    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" podUID="3a2a3118-97a9-4916-b7c3-724cc9d741af" containerName="main" containerID="cri-o://7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755" gracePeriod=45
Oct 30 12:09:06.838304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.838278    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:09:06.979398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979364    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir\") pod \"3a2a3118-97a9-4916-b7c3-724cc9d741af\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") "
Oct 30 12:09:06.979623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979408    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs\") pod \"3a2a3118-97a9-4916-b7c3-724cc9d741af\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") "
Oct 30 12:09:06.979623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979492    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache\") pod \"3a2a3118-97a9-4916-b7c3-724cc9d741af\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") "
Oct 30 12:09:06.979623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979577    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home\") pod \"3a2a3118-97a9-4916-b7c3-724cc9d741af\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") "
Oct 30 12:09:06.979805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw\" (UniqueName: \"kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw\") pod \"3a2a3118-97a9-4916-b7c3-724cc9d741af\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") "
Oct 30 12:09:06.979805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979727    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir" pod "3a2a3118-97a9-4916-b7c3-724cc9d741af" (UID: "3a2a3118-97a9-4916-b7c3-724cc9d741af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:06.979805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979764    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm\") pod \"3a2a3118-97a9-4916-b7c3-724cc9d741af\" (UID: \"3a2a3118-97a9-4916-b7c3-724cc9d741af\") "
Oct 30 12:09:06.979805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979788    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache" pod "3a2a3118-97a9-4916-b7c3-724cc9d741af" (UID: "3a2a3118-97a9-4916-b7c3-724cc9d741af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:06.980063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.979799    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home" pod "3a2a3118-97a9-4916-b7c3-724cc9d741af" (UID: "3a2a3118-97a9-4916-b7c3-724cc9d741af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:06.980504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.980401    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:06.980504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.980423    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:06.980504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.980432    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:06.981923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.981897    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs" pod "3a2a3118-97a9-4916-b7c3-724cc9d741af" (UID: "3a2a3118-97a9-4916-b7c3-724cc9d741af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:09:06.982026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.982008    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw" pod "3a2a3118-97a9-4916-b7c3-724cc9d741af" (UID: "3a2a3118-97a9-4916-b7c3-724cc9d741af"). InnerVolumeSpecName "kube-api-access-lllhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:06.982084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:06.982040    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm" pod "3a2a3118-97a9-4916-b7c3-724cc9d741af" (UID: "3a2a3118-97a9-4916-b7c3-724cc9d741af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:07.081702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.081622    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2a3118-97a9-4916-b7c3-724cc9d741af-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:07.081702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.081652    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lllhw\" (UniqueName: \"kubernetes.io/projected/3a2a3118-97a9-4916-b7c3-724cc9d741af-kube-api-access-lllhw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:07.081702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.081662    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2a3118-97a9-4916-b7c3-724cc9d741af-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:07.155210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.155176    5060 generic.go:356] "Generic (PLEG): container finished" podID="3a2a3118-97a9-4916-b7c3-724cc9d741af" containerID="7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755" exitCode=0
Oct 30 12:09:07.155722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.155219    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"
Oct 30 12:09:07.155722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.155264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" event={"ID":"3a2a3118-97a9-4916-b7c3-724cc9d741af","Type":"ContainerDied","Data":"7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755"}
Oct 30 12:09:07.155722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.155308    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" event={"ID":"3a2a3118-97a9-4916-b7c3-724cc9d741af","Type":"ContainerDied","Data":"277d114d64f6b8681d3c9988c8000a93536ef362e9a611ea8d61a217b8b47db6"}
Oct 30 12:09:07.155722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.155327    5060 scope.go:122] "RemoveContainer" containerID="7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755"
Oct 30 12:09:07.168413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.168394    5060 scope.go:122] "RemoveContainer" containerID="7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755"
Oct 30 12:09:07.168804 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:09:07.168775    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755\": container with ID starting with 7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755 not found: ID does not exist" containerID="7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755"
Oct 30 12:09:07.168921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.168809    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755"} err="failed to get container status \"7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755\": rpc error: code = NotFound desc = could not find container \"7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755\": container with ID starting with 7c3d384bed0eca190ae9d030b71cde7d8430add2205b1dd6e78cece8542a0755 not found: ID does not exist"
Oct 30 12:09:07.210371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.210335    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"]
Oct 30 12:09:07.221958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.221923    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks"]
Oct 30 12:09:07.834276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.834238    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-a51616/explicit-model-a51616-kserve-5bd6bd4cc5-r5sks" podUID="3a2a3118-97a9-4916-b7c3-724cc9d741af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.245:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:09:07.855924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:07.855891    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:08.336085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:08.336044    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2a3118-97a9-4916-b7c3-724cc9d741af" path="/var/lib/kubelet/pods/3a2a3118-97a9-4916-b7c3-724cc9d741af/volumes"
Oct 30 12:09:09.331550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:09.331521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:09.993357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:09.993319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:10.161987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:10.161938    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:10.173547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:10.173515    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"]
Oct 30 12:09:10.186829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:10.186800    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a51616-9zcbf"]
Oct 30 12:09:10.309997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:10.309901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:10.336811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:10.336780    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7346f194-38e3-48cb-bad3-f212fb736ae9" path="/var/lib/kubelet/pods/7346f194-38e3-48cb-bad3-f212fb736ae9/volumes"
Oct 30 12:09:10.713083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:10.713036    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:12.055897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:12.055864    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:13.825549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:13.825130    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:15.498947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:15.498907    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:17.032361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.032323    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"]
Oct 30 12:09:17.036536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.036506    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e97c23-gk7tz"]
Oct 30 12:09:17.273888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.273845    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:17.427812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.427715    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:17.575368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.575332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:17.765319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.765289    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"]
Oct 30 12:09:17.765580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.765550    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy" containerID="cri-o://efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0" gracePeriod=30
Oct 30 12:09:17.820806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:17.820772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:18.335769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:18.335733    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc559e6-52c2-404d-850b-0af5f67c07d7" path="/var/lib/kubelet/pods/5dc559e6-52c2-404d-850b-0af5f67c07d7/volumes"
Oct 30 12:09:18.486651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:18.486614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:18.981423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:18.981381    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:19.194479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:19.190918    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:19.860170 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:09:19.860129    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:09:20.921032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:20.921001    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:21.123111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:21.123079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:21.313101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:21.313064    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:22.952639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:22.952559    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:23.146881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.146854    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:09:23.246217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246180    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246307    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246353    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246416    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246516    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:23.246692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246540    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246681    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246775    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246891    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.246933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246911    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:23.246933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.246919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs\") pod \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\" (UID: \"10ec294a-c5b8-4c9f-b7ab-f6babf112b99\") "
Oct 30 12:09:23.247114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.247084    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:23.247537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.247515    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:23.247705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.247680    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.247869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.247855    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.247977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.247964    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.248274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.248256    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:23.250729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.250699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:09:23.270506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.270426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:23.270750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.270586    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "kube-api-access-66vh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:23.272278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.272248    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy" pod "10ec294a-c5b8-4c9f-b7ab-f6babf112b99" (UID: "10ec294a-c5b8-4c9f-b7ab-f6babf112b99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:23.349195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.349161    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66vh2\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-kube-api-access-66vh2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.349195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.349192    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.349195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.349202    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.349466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.349209    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.349466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.349217    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.349466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.349224    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ec294a-c5b8-4c9f-b7ab-f6babf112b99-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:23.370812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.370728    5060 generic.go:356] "Generic (PLEG): container finished" podID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerID="efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0" exitCode=0
Oct 30 12:09:23.370812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.370794    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"
Oct 30 12:09:23.371043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.370805    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" event={"ID":"10ec294a-c5b8-4c9f-b7ab-f6babf112b99","Type":"ContainerDied","Data":"efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0"}
Oct 30 12:09:23.371043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.370861    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2" event={"ID":"10ec294a-c5b8-4c9f-b7ab-f6babf112b99","Type":"ContainerDied","Data":"b86c83e25db12d2c98a66c9f1af4a02dc5f22af3d50aaac71e425f398a04e1e7"}
Oct 30 12:09:23.371043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.370883    5060 scope.go:122] "RemoveContainer" containerID="efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0"
Oct 30 12:09:23.382365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.382343    5060 scope.go:122] "RemoveContainer" containerID="456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"
Oct 30 12:09:23.393977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.393955    5060 scope.go:122] "RemoveContainer" containerID="efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0"
Oct 30 12:09:23.394310 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:09:23.394286    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0\": container with ID starting with efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0 not found: ID does not exist" containerID="efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0"
Oct 30 12:09:23.394354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.394320    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0"} err="failed to get container status \"efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0\": rpc error: code = NotFound desc = could not find container \"efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0\": container with ID starting with efd50c7a4c80976cfdfee333fc46bf9fc865af79c23322c13b5ea79c44dffda0 not found: ID does not exist"
Oct 30 12:09:23.394354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.394338    5060 scope.go:122] "RemoveContainer" containerID="456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"
Oct 30 12:09:23.394722 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:09:23.394697    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae\": container with ID starting with 456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae not found: ID does not exist" containerID="456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"
Oct 30 12:09:23.394811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.394728    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae"} err="failed to get container status \"456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae\": rpc error: code = NotFound desc = could not find container \"456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae\": container with ID starting with 456a6b2784de864d1bbf134b05e6464b5696d6b4da20adfcccb4ce3c78be20ae not found: ID does not exist"
Oct 30 12:09:23.438083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.438043    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"]
Oct 30 12:09:23.443098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.443068    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"]
Oct 30 12:09:23.444100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444078    5060 cpu_manager.go:414] "containerMap: removing container" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy"
Oct 30 12:09:23.444185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444103    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy"
Oct 30 12:09:23.444185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444121    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3a2a3118-97a9-4916-b7c3-724cc9d741af" containerName="main"
Oct 30 12:09:23.444185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444129    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2a3118-97a9-4916-b7c3-724cc9d741af" containerName="main"
Oct 30 12:09:23.444185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444180    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7346f194-38e3-48cb-bad3-f212fb736ae9" containerName="revoke-keys"
Oct 30 12:09:23.444385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444188    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7346f194-38e3-48cb-bad3-f212fb736ae9" containerName="revoke-keys"
Oct 30 12:09:23.444385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444344    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy"
Oct 30 12:09:23.444385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444357    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7346f194-38e3-48cb-bad3-f212fb736ae9" containerName="revoke-keys"
Oct 30 12:09:23.444385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444374    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy"
Oct 30 12:09:23.444385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.444384    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2a3118-97a9-4916-b7c3-724cc9d741af" containerName="main"
Oct 30 12:09:23.450266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.450239    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a51616-openshift-default-6556fbb557-2v8f2"]
Oct 30 12:09:23.450433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.450366    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.469657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.469618    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"]
Oct 30 12:09:23.470254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.470045    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-afbe5f\"/\"kube-root-ca.crt\""
Oct 30 12:09:23.470982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.470863    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-afbe5f\"/\"openshift-service-ca.crt\""
Oct 30 12:09:23.539830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.539786    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:23.553388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.553348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.553675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.553412    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.553675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.553568    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpg7\" (UniqueName: \"kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.553675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.553622    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.553824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.553751    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.553824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.553793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.558136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.558110    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-afbe5f\"/\"default-dockercfg-75d8s\""
Oct 30 12:09:23.558136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.558135    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-afbe5f\"/\"noten-model-afbe5f-kserve-self-signed-certs\""
Oct 30 12:09:23.655394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.655394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655377    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.655673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655571    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.655673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655641    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.655783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655696    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpg7\" (UniqueName: \"kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.655783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.655783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.655773    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.656182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.656152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.656296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.656182    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.658585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.658560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.659088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.659066    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.701481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.684198    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpg7\" (UniqueName: \"kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7\") pod \"noten-model-afbe5f-kserve-b4b94566f-lfsfr\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") " pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:23.792790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:23.786270    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:24.169316 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:09:24.169248    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda74b35bf_f67d_40cd_a690_0f6cca91a92a.slice/crio-a00d08accb7ab957ca337df5709ef468f0f21f7e7394690a26ec9392af64ddf9 WatchSource:0}: Error finding container a00d08accb7ab957ca337df5709ef468f0f21f7e7394690a26ec9392af64ddf9: Status 404 returned error can't find the container with id a00d08accb7ab957ca337df5709ef468f0f21f7e7394690a26ec9392af64ddf9
Oct 30 12:09:24.172530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.172500    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"]
Oct 30 12:09:24.338926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.338893    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" path="/var/lib/kubelet/pods/10ec294a-c5b8-4c9f-b7ab-f6babf112b99/volumes"
Oct 30 12:09:24.383368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.383326    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr" event={"ID":"a74b35bf-f67d-40cd-a690-0f6cca91a92a","Type":"ContainerStarted","Data":"b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47"}
Oct 30 12:09:24.383599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.383377    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr" event={"ID":"a74b35bf-f67d-40cd-a690-0f6cca91a92a","Type":"ContainerStarted","Data":"a00d08accb7ab957ca337df5709ef468f0f21f7e7394690a26ec9392af64ddf9"}
Oct 30 12:09:24.425328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.425209    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr" podStartSLOduration=1.4251894059999999 podStartE2EDuration="1.425189406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:24.424370954 +0000 UTC m=+2368.219255449" watchObservedRunningTime="2026-10-30 12:09:24.425189406 +0000 UTC m=+2368.220073900"
Oct 30 12:09:24.787145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.787106    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:24.843484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:24.843417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:26.692330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:26.692296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:27.032635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:27.032591    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"]
Oct 30 12:09:27.037757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:27.037730    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72ee423-hfxqb"]
Oct 30 12:09:28.336236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:28.336203    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e746fc-3bd6-4fe6-9d16-044c9848ec1c" path="/var/lib/kubelet/pods/c4e746fc-3bd6-4fe6-9d16-044c9848ec1c/volumes"
Oct 30 12:09:28.440915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:28.440886    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:30.149232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:30.149188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:31.922910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:31.922862    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:33.600834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:33.600792    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:33.787137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:33.787102    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:33.802558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:33.802524    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:34.480266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:34.480234    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:09:35.303056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:35.302912    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:36.999556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:36.999464    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"]
Oct 30 12:09:37.000040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:36.999770    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy" containerID="cri-o://a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f" gracePeriod=30
Oct 30 12:09:37.036906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:37.036873    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"]
Oct 30 12:09:37.040996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:37.040967    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355c016-97pkn"]
Oct 30 12:09:37.080101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:37.080061    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:38.335856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:38.335824    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a81023-9ef9-4246-955a-1cfec4895f55" path="/var/lib/kubelet/pods/42a81023-9ef9-4246-955a-1cfec4895f55/volumes"
Oct 30 12:09:38.873097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:38.873050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:40.641567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:40.641533    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:42.296785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.296758    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:09:42.430953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.430915    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.430962    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431034    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431129    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431186    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431258    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.431349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431331    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431433    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431464    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.431650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431547    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79\") pod \"01871d97-ee7d-44da-b5f6-a462c8826a94\" (UID: \"01871d97-ee7d-44da-b5f6-a462c8826a94\") "
Oct 30 12:09:42.431650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431554    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:42.431650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431558    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.431827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.431691    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.432533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.432507    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.432671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.432536    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.432671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.432553    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01871d97-ee7d-44da-b5f6-a462c8826a94-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.432671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.432566    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.432671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.432658    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.434020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.433989    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "kube-api-access-jkv79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:42.434156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.434140    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:09:42.434156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.434133    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.434255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.434160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token" pod "01871d97-ee7d-44da-b5f6-a462c8826a94" (UID: "01871d97-ee7d-44da-b5f6-a462c8826a94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:42.438163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.438135    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:42.534424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.534332    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.534424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.534364    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.534424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.534377    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkv79\" (UniqueName: \"kubernetes.io/projected/01871d97-ee7d-44da-b5f6-a462c8826a94-kube-api-access-jkv79\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.534424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.534391    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01871d97-ee7d-44da-b5f6-a462c8826a94-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.571000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.570970    5060 generic.go:356] "Generic (PLEG): container finished" podID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerID="a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f" exitCode=0
Oct 30 12:09:42.571213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.571021    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"
Oct 30 12:09:42.571213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.571057    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" event={"ID":"01871d97-ee7d-44da-b5f6-a462c8826a94","Type":"ContainerDied","Data":"a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f"}
Oct 30 12:09:42.571213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.571116    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7" event={"ID":"01871d97-ee7d-44da-b5f6-a462c8826a94","Type":"ContainerDied","Data":"5a5626a382d12a70e0081b593a65777f46ab4518c44a1fd195998b2e9e5d59f8"}
Oct 30 12:09:42.571213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.571139    5060 scope.go:122] "RemoveContainer" containerID="a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f"
Oct 30 12:09:42.583710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.583686    5060 scope.go:122] "RemoveContainer" containerID="a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f"
Oct 30 12:09:42.584121 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:09:42.584099    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f\": container with ID starting with a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f not found: ID does not exist" containerID="a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f"
Oct 30 12:09:42.584213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.584132    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f"} err="failed to get container status \"a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f\": rpc error: code = NotFound desc = could not find container \"a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f\": container with ID starting with a8e4a178eba6c0d45a35e95c0de97a32e47c4d74e20740f2f2ce5c072ac9f18f not found: ID does not exist"
Oct 30 12:09:42.649495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.649412    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"]
Oct 30 12:09:42.649495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:42.649489    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-69wm7"]
Oct 30 12:09:44.234032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:44.233995    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:44.336285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:44.336255    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" path="/var/lib/kubelet/pods/01871d97-ee7d-44da-b5f6-a462c8826a94/volumes"
Oct 30 12:09:45.180323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.180280    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:45.794082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.794040    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"]
Oct 30 12:09:45.795020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.795001    5060 cpu_manager.go:414] "containerMap: removing container" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy"
Oct 30 12:09:45.795111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.795021    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ec294a-c5b8-4c9f-b7ab-f6babf112b99" containerName="istio-proxy"
Oct 30 12:09:45.795111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.795045    5060 cpu_manager.go:414] "containerMap: removing container" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy"
Oct 30 12:09:45.795111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.795055    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy"
Oct 30 12:09:45.795275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.795208    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="01871d97-ee7d-44da-b5f6-a462c8826a94" containerName="istio-proxy"
Oct 30 12:09:45.800583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.800555    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:45.834652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.834595    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:09:45.834652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.834610    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:09:45.848726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.848697    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:09:45.848898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.848764    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:09:45.855043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.855012    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"]
Oct 30 12:09:45.985739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:45.985692    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:46.013795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.013714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq54q\" (UniqueName: \"kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.014373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.014334    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.014872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.014831    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.117101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.117002    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq54q\" (UniqueName: \"kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.117332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.117130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.117332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.117302    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.118064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.118038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.120165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.120144    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.140660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.140630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq54q\" (UniqueName: \"kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q\") pod \"maas-api-revoke-sub-39064b5d-mwntv\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.422617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.422536    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:46.741232 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:09:46.741197    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5cdeb1_67fa_4687_a3b8_4d5f8b9746d7.slice/crio-789962207e12b890f28bb15979564ec89ee7bc6f017a14193f6e8dfe7f28ac22 WatchSource:0}: Error finding container 789962207e12b890f28bb15979564ec89ee7bc6f017a14193f6e8dfe7f28ac22: Status 404 returned error can't find the container with id 789962207e12b890f28bb15979564ec89ee7bc6f017a14193f6e8dfe7f28ac22
Oct 30 12:09:46.742928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:46.742908    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"]
Oct 30 12:09:47.622633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:47.622603    5060 generic.go:356] "Generic (PLEG): container finished" podID="ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" containerID="2e3f4461b811d055d66fc7eeb2d6fa20501e60684c7233199bae30720d8b192b" exitCode=0
Oct 30 12:09:47.623015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:47.622652    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv" event={"ID":"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7","Type":"ContainerDied","Data":"2e3f4461b811d055d66fc7eeb2d6fa20501e60684c7233199bae30720d8b192b"}
Oct 30 12:09:47.623015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:47.622714    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv" event={"ID":"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7","Type":"ContainerStarted","Data":"789962207e12b890f28bb15979564ec89ee7bc6f017a14193f6e8dfe7f28ac22"}
Oct 30 12:09:47.683933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:47.679941    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:48.799382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.799352    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:48.858951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.858916    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q\" (UniqueName: \"kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q\") pod \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") "
Oct 30 12:09:48.859129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.858975    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca\") pod \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") "
Oct 30 12:09:48.859238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.859218    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca\") pod \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\" (UID: \"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7\") "
Oct 30 12:09:48.859446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.859412    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca" pod "ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" (UID: "ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:48.860107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.860078    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:48.861984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.861952    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca" pod "ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" (UID: "ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:09:48.861984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.861964    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q" pod "ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" (UID: "ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7"). InnerVolumeSpecName "kube-api-access-sq54q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:48.962080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.962050    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:48.962080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:48.962079    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq54q\" (UniqueName: \"kubernetes.io/projected/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7-kube-api-access-sq54q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:49.458927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:49.458892    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:49.646826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:49.646792    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"
Oct 30 12:09:49.646826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:49.646814    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv" event={"ID":"ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7","Type":"ContainerDied","Data":"789962207e12b890f28bb15979564ec89ee7bc6f017a14193f6e8dfe7f28ac22"}
Oct 30 12:09:49.647093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:49.646866    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789962207e12b890f28bb15979564ec89ee7bc6f017a14193f6e8dfe7f28ac22"
Oct 30 12:09:49.859568 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:09:49.859465    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:09:49.955935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:49.955896    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"]
Oct 30 12:09:50.012308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:50.012267    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:50.212324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:50.212289    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:50.540367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:50.540282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:50.740844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:50.740814    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:51.252302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:51.252260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:51.680181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:51.680063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:53.053105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:53.053062    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:54.761338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:54.761303    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:55.716695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:55.716659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:56.067466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.067352    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"]
Oct 30 12:09:56.068046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.068029    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" containerName="revoke-keys"
Oct 30 12:09:56.068117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.068047    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" containerName="revoke-keys"
Oct 30 12:09:56.068175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.068162    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" containerName="revoke-keys"
Oct 30 12:09:56.071842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.071820    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.094263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.094219    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"]
Oct 30 12:09:56.112179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.112140    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:09:56.112382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.112212    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:09:56.112382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.112260    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:09:56.112912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.112890    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:09:56.223565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.223525    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.223801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.223588    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.223919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.223897    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvrc\" (UniqueName: \"kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.325481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.325367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvrc\" (UniqueName: \"kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.325660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.325492    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.325660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.325522    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.328240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.328212    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.339588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.339554    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:09:56.346444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.346417    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.361056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.361026    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:09:56.370744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.370711    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvrc\" (UniqueName: \"kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc\") pod \"maas-api-revoke-sub-4f1c78a6-rprgg\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.408741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.408711    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:09:56.415114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.415086    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:56.648587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.648504    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:56.733713 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:09:56.733684    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eadb3b_8f57_45dc_b82c_89f154769450.slice/crio-eee02942c9517e94c49a008eb3b13082759351185c7285146eec70ff88054272 WatchSource:0}: Error finding container eee02942c9517e94c49a008eb3b13082759351185c7285146eec70ff88054272: Status 404 returned error can't find the container with id eee02942c9517e94c49a008eb3b13082759351185c7285146eec70ff88054272
Oct 30 12:09:56.734549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:56.734524    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"]
Oct 30 12:09:57.381548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:57.379149    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:09:57.395040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:57.395012    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:09:57.726035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:57.725990    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6eadb3b-8f57-45dc-b82c-89f154769450" containerID="d947386fe019235b3fd0bdb0db5c7580f559464a7150d72db1d499ba70f4c182" exitCode=0
Oct 30 12:09:57.726284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:57.726058    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg" event={"ID":"a6eadb3b-8f57-45dc-b82c-89f154769450","Type":"ContainerDied","Data":"d947386fe019235b3fd0bdb0db5c7580f559464a7150d72db1d499ba70f4c182"}
Oct 30 12:09:57.726284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:57.726098    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg" event={"ID":"a6eadb3b-8f57-45dc-b82c-89f154769450","Type":"ContainerStarted","Data":"eee02942c9517e94c49a008eb3b13082759351185c7285146eec70ff88054272"}
Oct 30 12:09:58.467227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:58.467184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:09:58.912794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:58.912766    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:59.063772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.063691    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca\") pod \"a6eadb3b-8f57-45dc-b82c-89f154769450\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") "
Oct 30 12:09:59.063772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.063736    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc\" (UniqueName: \"kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc\") pod \"a6eadb3b-8f57-45dc-b82c-89f154769450\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") "
Oct 30 12:09:59.064021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.063899    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca\") pod \"a6eadb3b-8f57-45dc-b82c-89f154769450\" (UID: \"a6eadb3b-8f57-45dc-b82c-89f154769450\") "
Oct 30 12:09:59.064271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.064245    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca" pod "a6eadb3b-8f57-45dc-b82c-89f154769450" (UID: "a6eadb3b-8f57-45dc-b82c-89f154769450"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:59.066051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.066025    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca" pod "a6eadb3b-8f57-45dc-b82c-89f154769450" (UID: "a6eadb3b-8f57-45dc-b82c-89f154769450"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:09:59.066051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.066037    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc" pod "a6eadb3b-8f57-45dc-b82c-89f154769450" (UID: "a6eadb3b-8f57-45dc-b82c-89f154769450"). InnerVolumeSpecName "kube-api-access-zgvrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:59.165504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.165469    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:59.165504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.165498    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgvrc\" (UniqueName: \"kubernetes.io/projected/a6eadb3b-8f57-45dc-b82c-89f154769450-kube-api-access-zgvrc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:59.165504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.165508    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6eadb3b-8f57-45dc-b82c-89f154769450-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:09:59.329197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.329106    5060 scope.go:122] "RemoveContainer" containerID="24d983a3840d42591cc48136f202d563e34ddad4b0258109a85d6ecc9f8ca535"
Oct 30 12:09:59.341422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.341395    5060 scope.go:122] "RemoveContainer" containerID="c90b5e27fd91962d8b2a9d2754fbb437cd3e0bff435a29ba3f7fa7acdbfcf53d"
Oct 30 12:09:59.354328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.354263    5060 scope.go:122] "RemoveContainer" containerID="4cf7a657eae3ea05daa2de49c3291fba19fb171f0fda7653539c689f8ed8d4e0"
Oct 30 12:09:59.367656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.367622    5060 scope.go:122] "RemoveContainer" containerID="102551bfd2310c4ef47b3ea879bb5e12998f1e8d058f5f9e89cb364f2ead5900"
Oct 30 12:09:59.379813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.379785    5060 scope.go:122] "RemoveContainer" containerID="112260326f49e509ebe71998345b828a9b706bab1719d9b65e07b1316f57e31f"
Oct 30 12:09:59.391040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.391012    5060 scope.go:122] "RemoveContainer" containerID="7fac8153452e850c46ff01f1ec31fedd3cb20ae760f5b244f3c3a744a14f71a9"
Oct 30 12:09:59.402283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.402264    5060 scope.go:122] "RemoveContainer" containerID="f1a565946910b96bfc534b62b310615c7e4ea144900240361a43029172ab7eae"
Oct 30 12:09:59.413558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.413534    5060 scope.go:122] "RemoveContainer" containerID="a9fdf14829eefe2f81a703909b9c07513b77adf279003dd426fec480dfb469d9"
Oct 30 12:09:59.424178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.424151    5060 scope.go:122] "RemoveContainer" containerID="5898f163744810e89dc0d29d20435e4b8d5ca4b1a80c5a505645653efad6f23b"
Oct 30 12:09:59.744752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.744721    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"
Oct 30 12:09:59.745186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.744752    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg" event={"ID":"a6eadb3b-8f57-45dc-b82c-89f154769450","Type":"ContainerDied","Data":"eee02942c9517e94c49a008eb3b13082759351185c7285146eec70ff88054272"}
Oct 30 12:09:59.745186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:09:59.744800    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eee02942c9517e94c49a008eb3b13082759351185c7285146eec70ff88054272"
Oct 30 12:10:00.170609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:00.170430    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:01.857002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:01.856966    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:03.641365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:03.638362    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:04.980482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:04.980411    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr" podUID="a74b35bf-f67d-40cd-a690-0f6cca91a92a" containerName="main" containerID="cri-o://b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47" gracePeriod=45
Oct 30 12:10:05.030521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.030481    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"]
Oct 30 12:10:05.042593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.042559    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dfec3b4-c6b6s"]
Oct 30 12:10:05.304093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.304062    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:10:05.362170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.362130    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home\") pod \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") "
Oct 30 12:10:05.362771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.362199    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs\") pod \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") "
Oct 30 12:10:05.362771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.362329    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7\" (UniqueName: \"kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7\") pod \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") "
Oct 30 12:10:05.362771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.362364    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir\") pod \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") "
Oct 30 12:10:05.362771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.362414    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache\") pod \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") "
Oct 30 12:10:05.362771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.362679    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm\") pod \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\" (UID: \"a74b35bf-f67d-40cd-a690-0f6cca91a92a\") "
Oct 30 12:10:05.363363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.363331    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home" pod "a74b35bf-f67d-40cd-a690-0f6cca91a92a" (UID: "a74b35bf-f67d-40cd-a690-0f6cca91a92a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.364281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.364254    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir" pod "a74b35bf-f67d-40cd-a690-0f6cca91a92a" (UID: "a74b35bf-f67d-40cd-a690-0f6cca91a92a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.364539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.364515    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache" pod "a74b35bf-f67d-40cd-a690-0f6cca91a92a" (UID: "a74b35bf-f67d-40cd-a690-0f6cca91a92a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.366208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.366185    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7" pod "a74b35bf-f67d-40cd-a690-0f6cca91a92a" (UID: "a74b35bf-f67d-40cd-a690-0f6cca91a92a"). InnerVolumeSpecName "kube-api-access-rjpg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:05.366541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.366517    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs" pod "a74b35bf-f67d-40cd-a690-0f6cca91a92a" (UID: "a74b35bf-f67d-40cd-a690-0f6cca91a92a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:10:05.366875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.366852    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm" pod "a74b35bf-f67d-40cd-a690-0f6cca91a92a" (UID: "a74b35bf-f67d-40cd-a690-0f6cca91a92a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.439482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.438857    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:05.464947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.464913    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjpg7\" (UniqueName: \"kubernetes.io/projected/a74b35bf-f67d-40cd-a690-0f6cca91a92a-kube-api-access-rjpg7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.465231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.464971    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.465231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.464986    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.465231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.464995    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.465231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.465002    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74b35bf-f67d-40cd-a690-0f6cca91a92a-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.465231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.465010    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74b35bf-f67d-40cd-a690-0f6cca91a92a-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.804469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.804426    5060 generic.go:356] "Generic (PLEG): container finished" podID="a74b35bf-f67d-40cd-a690-0f6cca91a92a" containerID="b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47" exitCode=0
Oct 30 12:10:05.804691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.804490    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"
Oct 30 12:10:05.804691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.804494    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr" event={"ID":"a74b35bf-f67d-40cd-a690-0f6cca91a92a","Type":"ContainerDied","Data":"b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47"}
Oct 30 12:10:05.804691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.804551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr" event={"ID":"a74b35bf-f67d-40cd-a690-0f6cca91a92a","Type":"ContainerDied","Data":"a00d08accb7ab957ca337df5709ef468f0f21f7e7394690a26ec9392af64ddf9"}
Oct 30 12:10:05.804691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.804575    5060 scope.go:122] "RemoveContainer" containerID="b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47"
Oct 30 12:10:05.820637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.820612    5060 scope.go:122] "RemoveContainer" containerID="b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47"
Oct 30 12:10:05.820992 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:05.820972    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47\": container with ID starting with b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47 not found: ID does not exist" containerID="b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47"
Oct 30 12:10:05.821083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.821000    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47"} err="failed to get container status \"b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47\": rpc error: code = NotFound desc = could not find container \"b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47\": container with ID starting with b2fdae951424846a2323057251415546c9aed4a895b04f01021e5d450020af47 not found: ID does not exist"
Oct 30 12:10:05.857946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.857917    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"]
Oct 30 12:10:05.869675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:05.869647    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-afbe5f/noten-model-afbe5f-kserve-b4b94566f-lfsfr"]
Oct 30 12:10:06.337266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:06.337234    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b777693-bd41-4fab-9cc1-137588803863" path="/var/lib/kubelet/pods/5b777693-bd41-4fab-9cc1-137588803863/volumes"
Oct 30 12:10:06.337702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:06.337665    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74b35bf-f67d-40cd-a690-0f6cca91a92a" path="/var/lib/kubelet/pods/a74b35bf-f67d-40cd-a690-0f6cca91a92a/volumes"
Oct 30 12:10:07.267132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:07.267084    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:07.867683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:07.867650    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:08.083099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:08.083067    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"]
Oct 30 12:10:08.110783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:08.110740    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65649434-h5j6l"]
Oct 30 12:10:08.117469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:08.117412    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:08.346957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:08.346922    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e4d31d-9879-4630-a4cc-3e5c00f578dc" path="/var/lib/kubelet/pods/36e4d31d-9879-4630-a4cc-3e5c00f578dc/volumes"
Oct 30 12:10:09.048527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:09.048490    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:09.841832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:09.838108    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"]
Oct 30 12:10:09.897085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:09.897037    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:10.843752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:10.843713    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:12.862962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:12.862925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:14.659723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:14.659684    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:16.456853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:16.456808    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:18.173217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:18.173187    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:19.543518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:19.543482    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"]
Oct 30 12:10:19.544033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:19.543709    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy" containerID="cri-o://437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c" gracePeriod=30
Oct 30 12:10:19.557552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:19.557511    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"]
Oct 30 12:10:19.557834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:19.557757    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy" containerID="cri-o://392672df93de39e5a52dbb991fefe87a37593308cbbf0a5b7670dd7ed22d38fe" gracePeriod=30
Oct 30 12:10:19.587567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:19.585491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:19.857300 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:19.857207    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:10:20.037250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:20.037209    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:21.787689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:21.787649    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:23.467895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.467859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:23.703858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.703813    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"]
Oct 30 12:10:23.704132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.704106    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy" containerID="cri-o://3ce51888a3239e8fd8c2be63705f25641e05804153d2cf184b14b30b5151fe31" gracePeriod=30
Oct 30 12:10:23.716290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.716254    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"]
Oct 30 12:10:23.716549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.716523    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy" containerID="cri-o://577010c93663ad858926ded8eb44276425b99914487f82e9606e4bb87611c6ec" gracePeriod=30
Oct 30 12:10:23.723360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.723278    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"]
Oct 30 12:10:23.723551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.723529    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy" containerID="cri-o://831a63c2501563e21a21ff8233383971bee583cac405323d20c7d052503b867e" gracePeriod=30
Oct 30 12:10:23.729295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.729259    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"]
Oct 30 12:10:23.729567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:23.729532    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy" containerID="cri-o://3d3de549e7c7bacaae76e675f46a24acefefefb074c8014076b55b6b9a000398" gracePeriod=30
Oct 30 12:10:24.873299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.873256    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" podUID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" containerName="main" containerID="cri-o://f078536878a7ceee17360d1e2c2f204f909e24d396c43ee95e3e96131dbb208a" gracePeriod=45
Oct 30 12:10:24.874576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.874549    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:10:24.891381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.891345    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" podUID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.238:8000/ready\": dial tcp 10.132.0.238:8000: connect: connection refused"
Oct 30 12:10:24.964096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964113    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964243    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964302    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964353    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964396    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964418    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964443    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964545    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert\") pod \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\" (UID: \"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78\") "
Oct 30 12:10:24.964919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964867    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:24.964919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.964881    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:24.965081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.965022    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:24.965211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.965193    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:24.965288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.965219    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:24.965288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.965240    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:24.965288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.965219    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:24.965722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.965397    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:24.967275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.967252    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:24.967275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.967265    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "kube-api-access-kcrwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:24.967633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.967614    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:24.967931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.967912    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy" pod "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" (UID: "5f4fcd11-c4ed-4bfd-b448-cef2afb67d78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:24.999688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.999660    5060 generic.go:356] "Generic (PLEG): container finished" podID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerID="437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c" exitCode=0
Oct 30 12:10:24.999854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.999711    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"
Oct 30 12:10:24.999854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.999765    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" event={"ID":"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78","Type":"ContainerDied","Data":"437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c"}
Oct 30 12:10:24.999854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.999818    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r" event={"ID":"5f4fcd11-c4ed-4bfd-b448-cef2afb67d78","Type":"ContainerDied","Data":"9bb5186a3a75aabb4fe4e5aa4131589a1ef19c13d3ad338c8f1a111533959a3f"}
Oct 30 12:10:24.999854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:24.999844    5060 scope.go:122] "RemoveContainer" containerID="437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c"
Oct 30 12:10:25.000172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.000141    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:10:25.002931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.002904    5060 generic.go:356] "Generic (PLEG): container finished" podID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerID="392672df93de39e5a52dbb991fefe87a37593308cbbf0a5b7670dd7ed22d38fe" exitCode=0
Oct 30 12:10:25.003079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.003056    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" event={"ID":"47eac8b2-20c1-49b3-a9e7-ff78c195a88a","Type":"ContainerDied","Data":"392672df93de39e5a52dbb991fefe87a37593308cbbf0a5b7670dd7ed22d38fe"}
Oct 30 12:10:25.006161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.006025    5060 generic.go:356] "Generic (PLEG): container finished" podID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" containerID="f078536878a7ceee17360d1e2c2f204f909e24d396c43ee95e3e96131dbb208a" exitCode=0
Oct 30 12:10:25.006161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.006117    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" event={"ID":"6024bc15-b94a-4f59-8342-9fc6109b8bd1","Type":"ContainerDied","Data":"f078536878a7ceee17360d1e2c2f204f909e24d396c43ee95e3e96131dbb208a"}
Oct 30 12:10:25.018342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.018321    5060 scope.go:122] "RemoveContainer" containerID="437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c"
Oct 30 12:10:25.019712 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:25.019689    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c\": container with ID starting with 437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c not found: ID does not exist" containerID="437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c"
Oct 30 12:10:25.019832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.019724    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c"} err="failed to get container status \"437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c\": rpc error: code = NotFound desc = could not find container \"437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c\": container with ID starting with 437eb32c3f16d89a2c39c5de8d025e0ecadd96fdf61344e201716e94687f982c not found: ID does not exist"
Oct 30 12:10:25.019832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.019748    5060 scope.go:122] "RemoveContainer" containerID="392672df93de39e5a52dbb991fefe87a37593308cbbf0a5b7670dd7ed22d38fe"
Oct 30 12:10:25.067419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.067027    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.067419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.067077    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.067419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.067092    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.067419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.067102    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.067419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.067113    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.067419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.067127    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrwf\" (UniqueName: \"kubernetes.io/projected/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78-kube-api-access-kcrwf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.106607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.106484    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"]
Oct 30 12:10:25.112748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.112715    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-72x6r"]
Oct 30 12:10:25.156113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.156087    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:10:25.167857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.167827    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.167901    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.167949    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168085    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168137    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.168287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168177    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168280    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168322    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.168351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168345    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168380    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168414    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token\") pod \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\" (UID: \"47eac8b2-20c1-49b3-a9e7-ff78c195a88a\") "
Oct 30 12:10:25.168518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168422    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.169370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.168659    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.169370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.169205    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:25.169852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.169815    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.169966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.169857    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.169966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.169871    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.169966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.169882    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.169966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.169894    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.177697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.176447    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:25.179575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.179548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:25.179725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.179550    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.179725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.179603    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn" pod "47eac8b2-20c1-49b3-a9e7-ff78c195a88a" (UID: "47eac8b2-20c1-49b3-a9e7-ff78c195a88a"). InnerVolumeSpecName "kube-api-access-5nxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:25.270714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.270684    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home\") pod \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") "
Oct 30 12:10:25.270899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.270784    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm\") pod \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") "
Oct 30 12:10:25.270899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.270812    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache\") pod \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") "
Oct 30 12:10:25.270899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.270838    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs\") pod \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") "
Oct 30 12:10:25.271175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.270920    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir\") pod \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") "
Oct 30 12:10:25.271175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.270995    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home" pod "6024bc15-b94a-4f59-8342-9fc6109b8bd1" (UID: "6024bc15-b94a-4f59-8342-9fc6109b8bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.271175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271034    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw\" (UniqueName: \"kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw\") pod \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\" (UID: \"6024bc15-b94a-4f59-8342-9fc6109b8bd1\") "
Oct 30 12:10:25.271175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271090    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache" pod "6024bc15-b94a-4f59-8342-9fc6109b8bd1" (UID: "6024bc15-b94a-4f59-8342-9fc6109b8bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.271385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271247    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir" pod "6024bc15-b94a-4f59-8342-9fc6109b8bd1" (UID: "6024bc15-b94a-4f59-8342-9fc6109b8bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.271707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271674    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxrn\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-kube-api-access-5nxrn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.271707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271699    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.271991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271714    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.271991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271727    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.271991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271739    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.271991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271751    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eac8b2-20c1-49b3-a9e7-ff78c195a88a-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.271991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.271763    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.273519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.273496    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm" pod "6024bc15-b94a-4f59-8342-9fc6109b8bd1" (UID: "6024bc15-b94a-4f59-8342-9fc6109b8bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:25.273613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.273516    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw" pod "6024bc15-b94a-4f59-8342-9fc6109b8bd1" (UID: "6024bc15-b94a-4f59-8342-9fc6109b8bd1"). InnerVolumeSpecName "kube-api-access-2p9sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:25.273731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.273711    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs" pod "6024bc15-b94a-4f59-8342-9fc6109b8bd1" (UID: "6024bc15-b94a-4f59-8342-9fc6109b8bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:10:25.355831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.355795    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:25.375579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.374838    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6024bc15-b94a-4f59-8342-9fc6109b8bd1-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.375579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.374895    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6024bc15-b94a-4f59-8342-9fc6109b8bd1-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:25.375579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:25.374910    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p9sw\" (UniqueName: \"kubernetes.io/projected/6024bc15-b94a-4f59-8342-9fc6109b8bd1-kube-api-access-2p9sw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:26.016124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.016090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j" event={"ID":"47eac8b2-20c1-49b3-a9e7-ff78c195a88a","Type":"ContainerDied","Data":"a95fe64023112baf159215823c952fa194694f1920a8ce145ceeca1d278540a3"}
Oct 30 12:10:26.016644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.016094    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"
Oct 30 12:10:26.018309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.018270    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv" event={"ID":"6024bc15-b94a-4f59-8342-9fc6109b8bd1","Type":"ContainerDied","Data":"c709806636d83a91691de20a39425f27e2f0ca3396805f781c88bc46f921add8"}
Oct 30 12:10:26.018498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.018325    5060 scope.go:122] "RemoveContainer" containerID="f078536878a7ceee17360d1e2c2f204f909e24d396c43ee95e3e96131dbb208a"
Oct 30 12:10:26.018570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.018496    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"
Oct 30 12:10:26.071025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.070991    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"]
Oct 30 12:10:26.083660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.083626    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-sm97j"]
Oct 30 12:10:26.113914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.113868    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"]
Oct 30 12:10:26.117683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.117654    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/auth-test-model-f1b14f-kserve-5c58677765-jkqbv"]
Oct 30 12:10:26.335927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.335851    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" path="/var/lib/kubelet/pods/47eac8b2-20c1-49b3-a9e7-ff78c195a88a/volumes"
Oct 30 12:10:26.336361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.336347    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" path="/var/lib/kubelet/pods/5f4fcd11-c4ed-4bfd-b448-cef2afb67d78/volumes"
Oct 30 12:10:26.336796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:26.336783    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" path="/var/lib/kubelet/pods/6024bc15-b94a-4f59-8342-9fc6109b8bd1/volumes"
Oct 30 12:10:27.064161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:27.064036    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:28.844263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:28.844220    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:29.102916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.101546    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"]
Oct 30 12:10:29.212507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.177991    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:29.318769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.318567    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerID="831a63c2501563e21a21ff8233383971bee583cac405323d20c7d052503b867e" exitCode=0
Oct 30 12:10:29.318769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.318687    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" event={"ID":"c4861f5c-0c88-4db7-9455-1a3832272d01","Type":"ContainerDied","Data":"831a63c2501563e21a21ff8233383971bee583cac405323d20c7d052503b867e"}
Oct 30 12:10:29.326175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.326123    5060 generic.go:356] "Generic (PLEG): container finished" podID="dcc40b82-8881-400c-813e-1f01248c02f1" containerID="3d3de549e7c7bacaae76e675f46a24acefefefb074c8014076b55b6b9a000398" exitCode=0
Oct 30 12:10:29.326518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.326493    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" event={"ID":"dcc40b82-8881-400c-813e-1f01248c02f1","Type":"ContainerDied","Data":"3d3de549e7c7bacaae76e675f46a24acefefefb074c8014076b55b6b9a000398"}
Oct 30 12:10:29.331816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.331785    5060 generic.go:356] "Generic (PLEG): container finished" podID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerID="3ce51888a3239e8fd8c2be63705f25641e05804153d2cf184b14b30b5151fe31" exitCode=0
Oct 30 12:10:29.332213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.332188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" event={"ID":"db7ec47f-8a5b-41fb-95e9-98096c41937b","Type":"ContainerDied","Data":"3ce51888a3239e8fd8c2be63705f25641e05804153d2cf184b14b30b5151fe31"}
Oct 30 12:10:29.334677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.334653    5060 generic.go:356] "Generic (PLEG): container finished" podID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerID="577010c93663ad858926ded8eb44276425b99914487f82e9606e4bb87611c6ec" exitCode=0
Oct 30 12:10:29.336608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.334848    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" event={"ID":"89c50c36-46a0-45a2-a11a-643ae80f040f","Type":"ContainerDied","Data":"577010c93663ad858926ded8eb44276425b99914487f82e9606e4bb87611c6ec"}
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.338768    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-fc21b2"]
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339668    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a74b35bf-f67d-40cd-a690-0f6cca91a92a" containerName="main"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339686    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74b35bf-f67d-40cd-a690-0f6cca91a92a" containerName="main"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339711    5060 cpu_manager.go:414] "containerMap: removing container" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339720    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339738    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339746    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339781    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6eadb3b-8f57-45dc-b82c-89f154769450" containerName="revoke-keys"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339788    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eadb3b-8f57-45dc-b82c-89f154769450" containerName="revoke-keys"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339804    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" containerName="main"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339811    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" containerName="main"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339984    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74b35bf-f67d-40cd-a690-0f6cca91a92a" containerName="main"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.339998    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="47eac8b2-20c1-49b3-a9e7-ff78c195a88a" containerName="istio-proxy"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.340008    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4fcd11-c4ed-4bfd-b448-cef2afb67d78" containerName="istio-proxy"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.340022    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eadb3b-8f57-45dc-b82c-89f154769450" containerName="revoke-keys"
Oct 30 12:10:29.341595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.340031    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6024bc15-b94a-4f59-8342-9fc6109b8bd1" containerName="main"
Oct 30 12:10:29.348130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.347022    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:29.369695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.368415    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\""
Oct 30 12:10:29.384996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.381382    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-fc21b2"]
Oct 30 12:10:29.415310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.415266    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmxh\" (UniqueName: \"kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh\") pod \"test-curl-7377-fc21b2\" (UID: \"3b567da8-ab81-4c82-ba59-4efeea964d9e\") " pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:29.519682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.517103    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmxh\" (UniqueName: \"kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh\") pod \"test-curl-7377-fc21b2\" (UID: \"3b567da8-ab81-4c82-ba59-4efeea964d9e\") " pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:29.532833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.530652    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:10:29.542266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.542210    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:10:29.550710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.548888    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmxh\" (UniqueName: \"kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh\") pod \"test-curl-7377-fc21b2\" (UID: \"3b567da8-ab81-4c82-ba59-4efeea964d9e\") " pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625300    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625398    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625429    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625498    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625545    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625603    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625645    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625678    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625706    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625776    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625877    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625942    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.625983    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.626031    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.626098    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert\") pod \"db7ec47f-8a5b-41fb-95e9-98096c41937b\" (UID: \"db7ec47f-8a5b-41fb-95e9-98096c41937b\") "
Oct 30 12:10:29.628580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.626130    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket\") pod \"c4861f5c-0c88-4db7-9455-1a3832272d01\" (UID: \"c4861f5c-0c88-4db7-9455-1a3832272d01\") "
Oct 30 12:10:29.629793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.628720    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.629793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.629039    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.629992    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.630855    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.631287    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.631769    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.632483    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.632916    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.633101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.642819    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.645643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.645501    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "kube-api-access-xvslr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.665789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.654324    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:29.666561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.666523    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.667048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.667027    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.671495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.671445    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:29.671881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.671847    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token" pod "db7ec47f-8a5b-41fb-95e9-98096c41937b" (UID: "db7ec47f-8a5b-41fb-95e9-98096c41937b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.694925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.694867    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.700066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.699961    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq" pod "c4861f5c-0c88-4db7-9455-1a3832272d01" (UID: "c4861f5c-0c88-4db7-9455-1a3832272d01"). InnerVolumeSpecName "kube-api-access-vpfbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.704826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.704796    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 12:10:29.707689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.707655    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:10:29.727760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727726    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727778    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727801    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727862    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727886    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727908    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727934    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.727977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.727975    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728019    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728110    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728137    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728184    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728217    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy\") pod \"dcc40b82-8881-400c-813e-1f01248c02f1\" (UID: \"dcc40b82-8881-400c-813e-1f01248c02f1\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728262    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728291    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.728471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data\") pod \"89c50c36-46a0-45a2-a11a-643ae80f040f\" (UID: \"89c50c36-46a0-45a2-a11a-643ae80f040f\") "
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728401    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728741    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728809    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.728994    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.729015    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.729238    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.729410    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:29.729614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.729418    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.730920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730895    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730926    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730943    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730959    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730972    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.730986    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731025    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731046    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc40b82-8881-400c-813e-1f01248c02f1-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731061    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731075    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731089    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4861f5c-0c88-4db7-9455-1a3832272d01-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731102    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731116    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731130    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c50c36-46a0-45a2-a11a-643ae80f040f-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731143    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ec47f-8a5b-41fb-95e9-98096c41937b-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731157    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731171    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvslr\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-kube-api-access-xvslr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731184    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731197    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpfbq\" (UniqueName: \"kubernetes.io/projected/c4861f5c-0c88-4db7-9455-1a3832272d01-kube-api-access-vpfbq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731208    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4861f5c-0c88-4db7-9455-1a3832272d01-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731217    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731226    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731237    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ec47f-8a5b-41fb-95e9-98096c41937b-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.731750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731545    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.732816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.731985    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.732816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.732172    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.732816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.732480    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:29.732816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.732636    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:29.732816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.732744    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "kube-api-access-7zpmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.732816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.732815    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.733607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.733584    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "kube-api-access-nnsw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.733868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.733831    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy" pod "dcc40b82-8881-400c-813e-1f01248c02f1" (UID: "dcc40b82-8881-400c-813e-1f01248c02f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:29.734446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.734431    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token" pod "89c50c36-46a0-45a2-a11a-643ae80f040f" (UID: "89c50c36-46a0-45a2-a11a-643ae80f040f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:29.832683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832654    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832683    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zpmk\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-kube-api-access-7zpmk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832698    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832710    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc40b82-8881-400c-813e-1f01248c02f1-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832723    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc40b82-8881-400c-813e-1f01248c02f1-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832736    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnsw4\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-kube-api-access-nnsw4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832750    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832763    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832776    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c50c36-46a0-45a2-a11a-643ae80f040f-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832787    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832798    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc40b82-8881-400c-813e-1f01248c02f1-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832809    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:29.832895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:29.832821    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c50c36-46a0-45a2-a11a-643ae80f040f-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:30.058745 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:10:30.058711    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b567da8_ab81_4c82_ba59_4efeea964d9e.slice/crio-30a95acb9aec46d8d0c088030a85d1280fa8d0b7f24498f7ec9e748465a6a722 WatchSource:0}: Error finding container 30a95acb9aec46d8d0c088030a85d1280fa8d0b7f24498f7ec9e748465a6a722: Status 404 returned error can't find the container with id 30a95acb9aec46d8d0c088030a85d1280fa8d0b7f24498f7ec9e748465a6a722
Oct 30 12:10:30.060563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.060539    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-fc21b2"]
Oct 30 12:10:30.347383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.347302    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"
Oct 30 12:10:30.348010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.347980    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949" event={"ID":"89c50c36-46a0-45a2-a11a-643ae80f040f","Type":"ContainerDied","Data":"aa9714d91620add251cf789834db2fda6965485eb112806d2d3bec7716c2a59f"}
Oct 30 12:10:30.348146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.348048    5060 scope.go:122] "RemoveContainer" containerID="577010c93663ad858926ded8eb44276425b99914487f82e9606e4bb87611c6ec"
Oct 30 12:10:30.350114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.350088    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-fc21b2" event={"ID":"3b567da8-ab81-4c82-ba59-4efeea964d9e","Type":"ContainerStarted","Data":"0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663"}
Oct 30 12:10:30.350242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.350129    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-fc21b2" event={"ID":"3b567da8-ab81-4c82-ba59-4efeea964d9e","Type":"ContainerStarted","Data":"30a95acb9aec46d8d0c088030a85d1280fa8d0b7f24498f7ec9e748465a6a722"}
Oct 30 12:10:30.353127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.353100    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9" event={"ID":"c4861f5c-0c88-4db7-9455-1a3832272d01","Type":"ContainerDied","Data":"67070b6da4a1aa404b00b818b19f9b0f40bcb998d71dea874fa4c76c0b143195"}
Oct 30 12:10:30.353339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.353224    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"
Oct 30 12:10:30.357437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.357305    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"
Oct 30 12:10:30.358702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.358582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx" event={"ID":"dcc40b82-8881-400c-813e-1f01248c02f1","Type":"ContainerDied","Data":"e137f83f1a5ba79045d5d08b51a7fbac89697cefcfabd7c749caba1748356611"}
Oct 30 12:10:30.371584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.371563    5060 scope.go:122] "RemoveContainer" containerID="831a63c2501563e21a21ff8233383971bee583cac405323d20c7d052503b867e"
Oct 30 12:10:30.371976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.371953    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc" event={"ID":"db7ec47f-8a5b-41fb-95e9-98096c41937b","Type":"ContainerDied","Data":"84209e8e103ced9e13768088c4275b3d51cd59266f6f6b6346e1157365b8133a"}
Oct 30 12:10:30.372087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.372072    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"
Oct 30 12:10:30.383094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.383073    5060 scope.go:122] "RemoveContainer" containerID="3d3de549e7c7bacaae76e675f46a24acefefefb074c8014076b55b6b9a000398"
Oct 30 12:10:30.393531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.393509    5060 scope.go:122] "RemoveContainer" containerID="3ce51888a3239e8fd8c2be63705f25641e05804153d2cf184b14b30b5151fe31"
Oct 30 12:10:30.408447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.408400    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-fc21b2" podStartSLOduration=1.408385524 podStartE2EDuration="1.408385524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:30.404695085 +0000 UTC m=+2434.199579575" watchObservedRunningTime="2026-10-30 12:10:30.408385524 +0000 UTC m=+2434.203270018"
Oct 30 12:10:30.472554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.472518    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"]
Oct 30 12:10:30.476433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.476406    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-ltfk9"]
Oct 30 12:10:30.510018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.509982    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"]
Oct 30 12:10:30.518797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.518763    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-jh8wc"]
Oct 30 12:10:30.552855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.552821    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"]
Oct 30 12:10:30.556723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.556694    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-t6949"]
Oct 30 12:10:30.592082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.592048    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"]
Oct 30 12:10:30.595827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.595799    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-4qjgx"]
Oct 30 12:10:30.661600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:30.661510    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:31.358481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.358267    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-fc21b2"]
Oct 30 12:10:31.363181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.363149    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-fc21b2"]
Oct 30 12:10:31.560416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.560380    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-705b28"]
Oct 30 12:10:31.560965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.560951    5060 cpu_manager.go:414] "containerMap: removing container" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy"
Oct 30 12:10:31.561013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.560966    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy"
Oct 30 12:10:31.561013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.560988    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" containerName="test-curl-7377-fc21b2"
Oct 30 12:10:31.561013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.560993    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" containerName="test-curl-7377-fc21b2"
Oct 30 12:10:31.561013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561003    5060 cpu_manager.go:414] "containerMap: removing container" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy"
Oct 30 12:10:31.561013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561007    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561018    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561023    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561035    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561039    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561127    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561136    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561147    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" containerName="istio-proxy"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561156    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" containerName="test-curl-7377-fc21b2"
Oct 30 12:10:31.561168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.561162    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" containerName="istio-proxy"
Oct 30 12:10:31.566153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.566134    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:31.581101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.581062    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-705b28"]
Oct 30 12:10:31.653596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.653513    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5dx\" (UniqueName: \"kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx\") pod \"test-curl-7377-705b28\" (UID: \"450405a7-686d-4658-86e1-38dc04b5cf58\") " pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:31.754857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.754828    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5dx\" (UniqueName: \"kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx\") pod \"test-curl-7377-705b28\" (UID: \"450405a7-686d-4658-86e1-38dc04b5cf58\") " pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:31.779226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.779199    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5dx\" (UniqueName: \"kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx\") pod \"test-curl-7377-705b28\" (UID: \"450405a7-686d-4658-86e1-38dc04b5cf58\") " pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:31.882724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:31.882689    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:32.186975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.186946    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-705b28"]
Oct 30 12:10:32.187658 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:10:32.187582    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod450405a7_686d_4658_86e1_38dc04b5cf58.slice/crio-df749c16d9a9e291f6f7c724fb23be8450574bf839a36bcf6f07ceba0e70dd99 WatchSource:0}: Error finding container df749c16d9a9e291f6f7c724fb23be8450574bf839a36bcf6f07ceba0e70dd99: Status 404 returned error can't find the container with id df749c16d9a9e291f6f7c724fb23be8450574bf839a36bcf6f07ceba0e70dd99
Oct 30 12:10:32.344520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.344439    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c50c36-46a0-45a2-a11a-643ae80f040f" path="/var/lib/kubelet/pods/89c50c36-46a0-45a2-a11a-643ae80f040f/volumes"
Oct 30 12:10:32.344989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.344974    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4861f5c-0c88-4db7-9455-1a3832272d01" path="/var/lib/kubelet/pods/c4861f5c-0c88-4db7-9455-1a3832272d01/volumes"
Oct 30 12:10:32.345407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.345385    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7ec47f-8a5b-41fb-95e9-98096c41937b" path="/var/lib/kubelet/pods/db7ec47f-8a5b-41fb-95e9-98096c41937b/volumes"
Oct 30 12:10:32.346335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.346311    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc40b82-8881-400c-813e-1f01248c02f1" path="/var/lib/kubelet/pods/dcc40b82-8881-400c-813e-1f01248c02f1/volumes"
Oct 30 12:10:32.368747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.368715    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:32.402980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.402925    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-705b28" event={"ID":"450405a7-686d-4658-86e1-38dc04b5cf58","Type":"ContainerStarted","Data":"068ce62814e6c3b683911b64e348d187c90d4c7be2f541f98522c8539ad288bb"}
Oct 30 12:10:32.402980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.402983    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-705b28" event={"ID":"450405a7-686d-4658-86e1-38dc04b5cf58","Type":"ContainerStarted","Data":"df749c16d9a9e291f6f7c724fb23be8450574bf839a36bcf6f07ceba0e70dd99"}
Oct 30 12:10:32.403242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.403215    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-fc21b2" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" containerName="test-curl-7377-fc21b2" containerID="cri-o://0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663" gracePeriod=2
Oct 30 12:10:32.444985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:32.444888    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-705b28" podStartSLOduration=1.444873588 podStartE2EDuration="1.444873588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:32.441391144 +0000 UTC m=+2436.236275635" watchObservedRunningTime="2026-10-30 12:10:32.444873588 +0000 UTC m=+2436.239758084"
Oct 30 12:10:34.250517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:34.250427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:34.621696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:34.621665    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:34.691697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:34.691665    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh\" (UniqueName: \"kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh\") pod \"3b567da8-ab81-4c82-ba59-4efeea964d9e\" (UID: \"3b567da8-ab81-4c82-ba59-4efeea964d9e\") "
Oct 30 12:10:34.693853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:34.693823    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh" pod "3b567da8-ab81-4c82-ba59-4efeea964d9e" (UID: "3b567da8-ab81-4c82-ba59-4efeea964d9e"). InnerVolumeSpecName "kube-api-access-ltmxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:34.793185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:34.793103    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmxh\" (UniqueName: \"kubernetes.io/projected/3b567da8-ab81-4c82-ba59-4efeea964d9e-kube-api-access-ltmxh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:35.427998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:35.427956    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b567da8-ab81-4c82-ba59-4efeea964d9e" containerID="0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663" exitCode=137
Oct 30 12:10:35.428507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:35.428011    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:35.428507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:35.428057    5060 scope.go:122] "RemoveContainer" containerID="0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663"
Oct 30 12:10:35.439998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:35.439978    5060 scope.go:122] "RemoveContainer" containerID="0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663"
Oct 30 12:10:35.440292 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:35.440268    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663\": container with ID starting with 0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663 not found: ID does not exist" containerID="0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663"
Oct 30 12:10:35.440356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:35.440301    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663"} err="failed to get container status \"0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663\": rpc error: code = NotFound desc = could not find container \"0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663\": container with ID starting with 0352d520dbf30cf3fb74669891b54023731a8cb83e230df2cf574e0679bd1663 not found: ID does not exist"
Oct 30 12:10:35.451364 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:35.451333    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-fc21b2\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:36.118149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:36.118084    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:36.344710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:36.344664    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" path="/var/lib/kubelet/pods/3b567da8-ab81-4c82-ba59-4efeea964d9e/volumes"
Oct 30 12:10:36.345154 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:36.345119    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-fc21b2\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="3b567da8-ab81-4c82-ba59-4efeea964d9e" pod="opendatahub/test-curl-7377-fc21b2"
Oct 30 12:10:36.353652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:36.353620    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"]
Oct 30 12:10:36.353904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:36.353835    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy" containerID="cri-o://0ef3e57a119d5b3192a16866f58fac2eeed110de8249dafef8c84290997b59bd" gracePeriod=30
Oct 30 12:10:37.983420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:37.983381    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:39.761040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:39.761007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:41.481112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.481075    5060 generic.go:356] "Generic (PLEG): container finished" podID="76c93c71-1617-4599-8f89-e30501be84b4" containerID="0ef3e57a119d5b3192a16866f58fac2eeed110de8249dafef8c84290997b59bd" exitCode=0
Oct 30 12:10:41.481589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.481148    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" event={"ID":"76c93c71-1617-4599-8f89-e30501be84b4","Type":"ContainerDied","Data":"0ef3e57a119d5b3192a16866f58fac2eeed110de8249dafef8c84290997b59bd"}
Oct 30 12:10:41.500465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.500410    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:41.664092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.664063    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:10:41.777247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777208    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777279    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777325    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777443    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777523    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777568    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:41.777750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777590    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777653    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.777750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777706    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket\") pod \"76c93c71-1617-4599-8f89-e30501be84b4\" (UID: \"76c93c71-1617-4599-8f89-e30501be84b4\") "
Oct 30 12:10:41.778037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.777766    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:41.778107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778088    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:41.778171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778112    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:41.778240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778166    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:41.778304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778273    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.778304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778293    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.778534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778332    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c93c71-1617-4599-8f89-e30501be84b4-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.778534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778345    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.778534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.778358    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.780133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.780105    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:41.780536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.780512    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "kube-api-access-bm9s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:41.780536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.780544    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:41.781158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.781134    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy" pod "76c93c71-1617-4599-8f89-e30501be84b4" (UID: "76c93c71-1617-4599-8f89-e30501be84b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:41.879533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.879497    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c93c71-1617-4599-8f89-e30501be84b4-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.879533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.879535    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.879754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.879551    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c93c71-1617-4599-8f89-e30501be84b4-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:41.879754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:41.879564    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9s9\" (UniqueName: \"kubernetes.io/projected/76c93c71-1617-4599-8f89-e30501be84b4-kube-api-access-bm9s9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:42.492946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:42.492912    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"
Oct 30 12:10:42.492946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:42.492938    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr" event={"ID":"76c93c71-1617-4599-8f89-e30501be84b4","Type":"ContainerDied","Data":"922e477753e18373a1e1209695a0741084d5e0ec825c8474ca53cf366246013c"}
Oct 30 12:10:42.493529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:42.493008    5060 scope.go:122] "RemoveContainer" containerID="0ef3e57a119d5b3192a16866f58fac2eeed110de8249dafef8c84290997b59bd"
Oct 30 12:10:42.559266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:42.559236    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"]
Oct 30 12:10:42.571410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:42.571384    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-znbdr"]
Oct 30 12:10:43.313975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.313929    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:43.475878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.475848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:43.489641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.489603    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-705b28"]
Oct 30 12:10:43.489859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.489832    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-705b28" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" containerName="test-curl-7377-705b28" containerID="cri-o://068ce62814e6c3b683911b64e348d187c90d4c7be2f541f98522c8539ad288bb" gracePeriod=2
Oct 30 12:10:43.494142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.492470    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-705b28"]
Oct 30 12:10:43.504065 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:43.504026    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-705b28\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:43.640146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640072    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-d48c42"]
Oct 30 12:10:43.640690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640675    5060 cpu_manager.go:414] "containerMap: removing container" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy"
Oct 30 12:10:43.640752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640690    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy"
Oct 30 12:10:43.640752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640699    5060 cpu_manager.go:414] "containerMap: removing container" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" containerName="test-curl-7377-705b28"
Oct 30 12:10:43.640752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640704    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" containerName="test-curl-7377-705b28"
Oct 30 12:10:43.640855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640813    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c93c71-1617-4599-8f89-e30501be84b4" containerName="istio-proxy"
Oct 30 12:10:43.640855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.640822    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" containerName="test-curl-7377-705b28"
Oct 30 12:10:43.645646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.645630    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:43.659250 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:43.659219    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-705b28\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:43.660071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.660050    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-d48c42"]
Oct 30 12:10:43.702914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.702878    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpfg\" (UniqueName: \"kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg\") pod \"test-curl-7377-d48c42\" (UID: \"832c3e1f-8f44-480b-a830-a06b38d6fa01\") " pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:43.804119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.804069    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpfg\" (UniqueName: \"kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg\") pod \"test-curl-7377-d48c42\" (UID: \"832c3e1f-8f44-480b-a830-a06b38d6fa01\") " pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:43.828751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.828716    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpfg\" (UniqueName: \"kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg\") pod \"test-curl-7377-d48c42\" (UID: \"832c3e1f-8f44-480b-a830-a06b38d6fa01\") " pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:43.869476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.860818    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"]
Oct 30 12:10:43.869476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.866194    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:43.898546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.894738    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f8f132-openshift-default-dockercfg-4nsfs\""
Oct 30 12:10:43.906361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.906322    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"]
Oct 30 12:10:43.930502 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:43.930420    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-705b28\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:43.946550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.946486    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:43.960237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:43.960203    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010248    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010309    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010400    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbr4\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010441    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010487    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010513    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010570    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010597    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.021869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.010662    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117028    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117191    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117221    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117303    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbr4\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117343    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117365    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117392    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117468    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.117494    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.118846    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.119076    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.121198    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.121832    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.122056    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.123647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.122535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.149020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.148922    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.149838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.149791    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbr4\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.156014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.155973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.199879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.199844    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:44.287265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.287182    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:44.353495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.352273    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c93c71-1617-4599-8f89-e30501be84b4" path="/var/lib/kubelet/pods/76c93c71-1617-4599-8f89-e30501be84b4/volumes"
Oct 30 12:10:44.386574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.385750    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg" podUID="215f0aaf-a182-4548-ad6d-aac08bde0a60" containerName="main" containerID="cri-o://185d197fe3958cf8451978df60a71604b8b1c547a3f251cd3d124239901dbaf0" gracePeriod=45
Oct 30 12:10:44.468740 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:10:44.468710    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod832c3e1f_8f44_480b_a830_a06b38d6fa01.slice/crio-aa0e57bb9360cc6c62314fec987d5732f8dafa78f6f19bcd92c2e6f1690c1865 WatchSource:0}: Error finding container aa0e57bb9360cc6c62314fec987d5732f8dafa78f6f19bcd92c2e6f1690c1865: Status 404 returned error can't find the container with id aa0e57bb9360cc6c62314fec987d5732f8dafa78f6f19bcd92c2e6f1690c1865
Oct 30 12:10:44.469673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.469650    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-d48c42"]
Oct 30 12:10:44.540285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.540242    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-d48c42" event={"ID":"832c3e1f-8f44-480b-a830-a06b38d6fa01","Type":"ContainerStarted","Data":"aa0e57bb9360cc6c62314fec987d5732f8dafa78f6f19bcd92c2e6f1690c1865"}
Oct 30 12:10:44.543111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.543081    5060 generic.go:356] "Generic (PLEG): container finished" podID="215f0aaf-a182-4548-ad6d-aac08bde0a60" containerID="185d197fe3958cf8451978df60a71604b8b1c547a3f251cd3d124239901dbaf0" exitCode=0
Oct 30 12:10:44.543331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.543264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg" event={"ID":"215f0aaf-a182-4548-ad6d-aac08bde0a60","Type":"ContainerDied","Data":"185d197fe3958cf8451978df60a71604b8b1c547a3f251cd3d124239901dbaf0"}
Oct 30 12:10:44.635821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.634377    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:10:44.635821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.634518    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:10:44.635821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.634563    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:10:44.636169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.636141    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"]
Oct 30 12:10:44.688011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.687986    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.836439    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache\") pod \"215f0aaf-a182-4548-ad6d-aac08bde0a60\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") "
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.836583    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm\") pod \"215f0aaf-a182-4548-ad6d-aac08bde0a60\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") "
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.836674    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h\" (UniqueName: \"kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h\") pod \"215f0aaf-a182-4548-ad6d-aac08bde0a60\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") "
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.836718    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home\") pod \"215f0aaf-a182-4548-ad6d-aac08bde0a60\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") "
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.836781    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs\" (UniqueName: \"kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs\") pod \"215f0aaf-a182-4548-ad6d-aac08bde0a60\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") "
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.836878    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir\") pod \"215f0aaf-a182-4548-ad6d-aac08bde0a60\" (UID: \"215f0aaf-a182-4548-ad6d-aac08bde0a60\") "
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.837299    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache" pod "215f0aaf-a182-4548-ad6d-aac08bde0a60" (UID: "215f0aaf-a182-4548-ad6d-aac08bde0a60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.837628    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home" pod "215f0aaf-a182-4548-ad6d-aac08bde0a60" (UID: "215f0aaf-a182-4548-ad6d-aac08bde0a60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.837636    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:44.838768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.837844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir" pod "215f0aaf-a182-4548-ad6d-aac08bde0a60" (UID: "215f0aaf-a182-4548-ad6d-aac08bde0a60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:44.841354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.840531    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs" pod "215f0aaf-a182-4548-ad6d-aac08bde0a60" (UID: "215f0aaf-a182-4548-ad6d-aac08bde0a60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:10:44.842144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.841612    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm" pod "215f0aaf-a182-4548-ad6d-aac08bde0a60" (UID: "215f0aaf-a182-4548-ad6d-aac08bde0a60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:44.842446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.842426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h" pod "215f0aaf-a182-4548-ad6d-aac08bde0a60" (UID: "215f0aaf-a182-4548-ad6d-aac08bde0a60"). InnerVolumeSpecName "kube-api-access-pxv2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:44.939122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.939039    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:44.939122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.939072    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxv2h\" (UniqueName: \"kubernetes.io/projected/215f0aaf-a182-4548-ad6d-aac08bde0a60-kube-api-access-pxv2h\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:44.939122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.939082    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:44.939122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.939090    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215f0aaf-a182-4548-ad6d-aac08bde0a60-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:44.939122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:44.939120    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215f0aaf-a182-4548-ad6d-aac08bde0a60-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:45.149198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.149160    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:45.554244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.554198    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-d48c42" event={"ID":"832c3e1f-8f44-480b-a830-a06b38d6fa01","Type":"ContainerStarted","Data":"b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb"}
Oct 30 12:10:45.556137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.556104    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" event={"ID":"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9","Type":"ContainerStarted","Data":"7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"}
Oct 30 12:10:45.556281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.556167    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" event={"ID":"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9","Type":"ContainerStarted","Data":"34f91a5a45634b32c2a8b128c709551702fd16f174f3df3796509fcbf40434f1"}
Oct 30 12:10:45.557954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.557933    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"
Oct 30 12:10:45.558069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.557971    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg" event={"ID":"215f0aaf-a182-4548-ad6d-aac08bde0a60","Type":"ContainerDied","Data":"d31ee9ce8124de1feb449146e8d9c0678f301d7bd742c9318bc61e94c36614c0"}
Oct 30 12:10:45.558069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.558018    5060 scope.go:122] "RemoveContainer" containerID="185d197fe3958cf8451978df60a71604b8b1c547a3f251cd3d124239901dbaf0"
Oct 30 12:10:45.561548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.561521    5060 generic.go:356] "Generic (PLEG): container finished" podID="450405a7-686d-4658-86e1-38dc04b5cf58" containerID="068ce62814e6c3b683911b64e348d187c90d4c7be2f541f98522c8539ad288bb" exitCode=137
Oct 30 12:10:45.597149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.597082    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-d48c42" podStartSLOduration=2.59706211 podStartE2EDuration="2.59706211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:45.59160874 +0000 UTC m=+2449.386493237" watchObservedRunningTime="2026-10-30 12:10:45.59706211 +0000 UTC m=+2449.391946605"
Oct 30 12:10:45.650143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.650085    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podStartSLOduration=2.650065315 podStartE2EDuration="2.650065315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:45.641518939 +0000 UTC m=+2449.436403435" watchObservedRunningTime="2026-10-30 12:10:45.650065315 +0000 UTC m=+2449.444949809"
Oct 30 12:10:45.682413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.682355    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"]
Oct 30 12:10:45.688243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.688211    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/auth-test-model-4fff4f-kserve-6445cc7ddf-2hgqg"]
Oct 30 12:10:45.738374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.738347    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:45.803089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.802996    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:45.848859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.848820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx\" (UniqueName: \"kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx\") pod \"450405a7-686d-4658-86e1-38dc04b5cf58\" (UID: \"450405a7-686d-4658-86e1-38dc04b5cf58\") "
Oct 30 12:10:45.851389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.851360    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx" pod "450405a7-686d-4658-86e1-38dc04b5cf58" (UID: "450405a7-686d-4658-86e1-38dc04b5cf58"). InnerVolumeSpecName "kube-api-access-hr5dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:45.946617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.946579    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:45.951189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:45.951150    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5dx\" (UniqueName: \"kubernetes.io/projected/450405a7-686d-4658-86e1-38dc04b5cf58-kube-api-access-hr5dx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:46.200736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.200704    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:46.206912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.206887    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:46.337401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.337371    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215f0aaf-a182-4548-ad6d-aac08bde0a60" path="/var/lib/kubelet/pods/215f0aaf-a182-4548-ad6d-aac08bde0a60/volumes"
Oct 30 12:10:46.337998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.337973    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450405a7-686d-4658-86e1-38dc04b5cf58" path="/var/lib/kubelet/pods/450405a7-686d-4658-86e1-38dc04b5cf58/volumes"
Oct 30 12:10:46.516612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.516512    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-d48c42"]
Oct 30 12:10:46.521741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.521703    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-d48c42"]
Oct 30 12:10:46.571960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.571933    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-705b28"
Oct 30 12:10:46.571960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.571956    5060 scope.go:122] "RemoveContainer" containerID="068ce62814e6c3b683911b64e348d187c90d4c7be2f541f98522c8539ad288bb"
Oct 30 12:10:46.572600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.572577    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:46.573777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.573698    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:10:46.670571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.670535    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-e41327"]
Oct 30 12:10:46.671118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.671102    5060 cpu_manager.go:414] "containerMap: removing container" podUID="832c3e1f-8f44-480b-a830-a06b38d6fa01" containerName="test-curl-7377-d48c42"
Oct 30 12:10:46.671118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.671118    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832c3e1f-8f44-480b-a830-a06b38d6fa01" containerName="test-curl-7377-d48c42"
Oct 30 12:10:46.671264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.671137    5060 cpu_manager.go:414] "containerMap: removing container" podUID="215f0aaf-a182-4548-ad6d-aac08bde0a60" containerName="main"
Oct 30 12:10:46.671264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.671142    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215f0aaf-a182-4548-ad6d-aac08bde0a60" containerName="main"
Oct 30 12:10:46.671264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.671238    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="832c3e1f-8f44-480b-a830-a06b38d6fa01" containerName="test-curl-7377-d48c42"
Oct 30 12:10:46.671264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.671249    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="215f0aaf-a182-4548-ad6d-aac08bde0a60" containerName="main"
Oct 30 12:10:46.676237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.676209    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:46.689810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.689780    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-e41327"]
Oct 30 12:10:46.762035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.761994    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744tc\" (UniqueName: \"kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc\") pod \"test-curl-7377-e41327\" (UID: \"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed\") " pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:46.863261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.863142    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744tc\" (UniqueName: \"kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc\") pod \"test-curl-7377-e41327\" (UID: \"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed\") " pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:46.886877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.886845    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-744tc\" (UniqueName: \"kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc\") pod \"test-curl-7377-e41327\" (UID: \"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed\") " pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:46.959013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.958958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:46.994253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:46.994214    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:47.355901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:47.355862    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-e41327"]
Oct 30 12:10:47.362911 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:10:47.362881    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aeb5054_03c7_48d1_8b47_59cb2cd2cbed.slice/crio-f300135f150a8b39e961e54aea267b03a00792de539640b42e1dc6db6c17b5cd WatchSource:0}: Error finding container f300135f150a8b39e961e54aea267b03a00792de539640b42e1dc6db6c17b5cd: Status 404 returned error can't find the container with id f300135f150a8b39e961e54aea267b03a00792de539640b42e1dc6db6c17b5cd
Oct 30 12:10:47.581654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:47.581614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-e41327" event={"ID":"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed","Type":"ContainerStarted","Data":"18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f"}
Oct 30 12:10:47.582121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:47.581670    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-e41327" event={"ID":"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed","Type":"ContainerStarted","Data":"f300135f150a8b39e961e54aea267b03a00792de539640b42e1dc6db6c17b5cd"}
Oct 30 12:10:47.582962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:47.582928    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-d48c42" podUID="832c3e1f-8f44-480b-a830-a06b38d6fa01" containerName="test-curl-7377-d48c42" containerID="cri-o://b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb" gracePeriod=2
Oct 30 12:10:47.619645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:47.619534    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-e41327" podStartSLOduration=1.619519843 podStartE2EDuration="1.619519843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:47.619335308 +0000 UTC m=+2451.414219799" watchObservedRunningTime="2026-10-30 12:10:47.619519843 +0000 UTC m=+2451.414404336"
Oct 30 12:10:47.633272 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:47.633232    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-d48c42\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="832c3e1f-8f44-480b-a830-a06b38d6fa01" pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:48.136284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.136179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:48.290525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.290483    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:48.520048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.520011    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-e41327"]
Oct 30 12:10:48.525430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.525394    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-e41327"]
Oct 30 12:10:48.671398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.671362    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-c7fbc8"]
Oct 30 12:10:48.672006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.671991    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" containerName="test-curl-7377-e41327"
Oct 30 12:10:48.672061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.672007    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" containerName="test-curl-7377-e41327"
Oct 30 12:10:48.672112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.672103    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" containerName="test-curl-7377-e41327"
Oct 30 12:10:48.675218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.675202    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:10:48.694025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.693999    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-c7fbc8"]
Oct 30 12:10:48.790713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.790621    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xmg\" (UniqueName: \"kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg\") pod \"test-curl-7377-c7fbc8\" (UID: \"a2f503d0-2921-49a3-918b-5ca8682a4339\") " pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:10:48.892570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.892523    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xmg\" (UniqueName: \"kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg\") pod \"test-curl-7377-c7fbc8\" (UID: \"a2f503d0-2921-49a3-918b-5ca8682a4339\") " pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:10:48.916425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.916395    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xmg\" (UniqueName: \"kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg\") pod \"test-curl-7377-c7fbc8\" (UID: \"a2f503d0-2921-49a3-918b-5ca8682a4339\") " pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:10:48.972576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.972532    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:48.987959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:48.987918    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:10:49.305995 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:10:49.305930    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f503d0_2921_49a3_918b_5ca8682a4339.slice/crio-28d63fdd7d42a74943ae9c7f1a19abb7c6d41bd446912e581bb901b532436411 WatchSource:0}: Error finding container 28d63fdd7d42a74943ae9c7f1a19abb7c6d41bd446912e581bb901b532436411: Status 404 returned error can't find the container with id 28d63fdd7d42a74943ae9c7f1a19abb7c6d41bd446912e581bb901b532436411
Oct 30 12:10:49.307917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.307890    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-c7fbc8"]
Oct 30 12:10:49.603951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.603920    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-c7fbc8" event={"ID":"a2f503d0-2921-49a3-918b-5ca8682a4339","Type":"ContainerStarted","Data":"dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78"}
Oct 30 12:10:49.604113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.603963    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-c7fbc8" event={"ID":"a2f503d0-2921-49a3-918b-5ca8682a4339","Type":"ContainerStarted","Data":"28d63fdd7d42a74943ae9c7f1a19abb7c6d41bd446912e581bb901b532436411"}
Oct 30 12:10:49.604113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.604015    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-e41327" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" containerName="test-curl-7377-e41327" containerID="cri-o://18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f" gracePeriod=2
Oct 30 12:10:49.617760 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:49.617722    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-e41327\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:49.658329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.658274    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-c7fbc8" podStartSLOduration=1.658258643 podStartE2EDuration="1.658258643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:49.655849523 +0000 UTC m=+2453.450734030" watchObservedRunningTime="2026-10-30 12:10:49.658258643 +0000 UTC m=+2453.453143137"
Oct 30 12:10:49.669464 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:49.669422    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-e41327\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:49.802069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.802033    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:49.816270 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:49.816234    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-e41327\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:49.858961 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:49.858872    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:10:49.906613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.906576    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg\" (UniqueName: \"kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg\") pod \"832c3e1f-8f44-480b-a830-a06b38d6fa01\" (UID: \"832c3e1f-8f44-480b-a830-a06b38d6fa01\") "
Oct 30 12:10:49.908898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.908865    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg" pod "832c3e1f-8f44-480b-a830-a06b38d6fa01" (UID: "832c3e1f-8f44-480b-a830-a06b38d6fa01"). InnerVolumeSpecName "kube-api-access-zgpfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:49.983362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:49.983154    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:50.013566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.008622    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpfg\" (UniqueName: \"kubernetes.io/projected/832c3e1f-8f44-480b-a830-a06b38d6fa01-kube-api-access-zgpfg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:50.134308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.134230    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:50.166196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.158944    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"]
Oct 30 12:10:50.175363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.174854    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.203092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.187610    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"]
Oct 30 12:10:50.203092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.199822    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f8f132-dockercfg-g84ld\""
Oct 30 12:10:50.203092 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:50.199916    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-e41327\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:50.203092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.200244    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f8f132\""
Oct 30 12:10:50.216617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.215897    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"]
Oct 30 12:10:50.216617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.216533    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.226475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.226377    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"]
Oct 30 12:10:50.274827 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:50.258339    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-e41327\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:50.319387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.319028    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.319387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.319100    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957q2\" (UniqueName: \"kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2\") pod \"payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.319387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.319140    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2dg\" (UniqueName: \"kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg\") pod \"payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.319387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.319174    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.336831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.336796    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832c3e1f-8f44-480b-a830-a06b38d6fa01" path="/var/lib/kubelet/pods/832c3e1f-8f44-480b-a830-a06b38d6fa01/volumes"
Oct 30 12:10:50.421502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.421387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.421679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.421520    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957q2\" (UniqueName: \"kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2\") pod \"payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.421679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.421627    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2dg\" (UniqueName: \"kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg\") pod \"payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.421800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.421714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.422279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.422256    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.422353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.422335    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.444867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.444834    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-957q2\" (UniqueName: \"kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2\") pod \"payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.445012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.444922    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2dg\" (UniqueName: \"kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg\") pod \"payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.517107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.517075    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:50.566696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.566652    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:50.618631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.618599    5060 generic.go:356] "Generic (PLEG): container finished" podID="832c3e1f-8f44-480b-a830-a06b38d6fa01" containerID="b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb" exitCode=137
Oct 30 12:10:50.619763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.618753    5060 scope.go:122] "RemoveContainer" containerID="b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb"
Oct 30 12:10:50.619763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.618814    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-d48c42"
Oct 30 12:10:50.643630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.641292    5060 scope.go:122] "RemoveContainer" containerID="b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb"
Oct 30 12:10:50.652328 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:50.651967    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb\": container with ID starting with b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb not found: ID does not exist" containerID="b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb"
Oct 30 12:10:50.652328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.652018    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb"} err="failed to get container status \"b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb\": rpc error: code = NotFound desc = could not find container \"b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb\": container with ID starting with b9d5472f50133639d27f08c6e0b1d9d4fbdaef103eb2e070cd8d9d16599d21bb not found: ID does not exist"
Oct 30 12:10:50.832523 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:10:50.832495    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88864962_1518_4fec_82dc_a65b4afe8214.slice/crio-aa026732bd36099bf6dd28c5c78cbb98ab0f2708e4785fb1b340c788c23885ba WatchSource:0}: Error finding container aa026732bd36099bf6dd28c5c78cbb98ab0f2708e4785fb1b340c788c23885ba: Status 404 returned error can't find the container with id aa026732bd36099bf6dd28c5c78cbb98ab0f2708e4785fb1b340c788c23885ba
Oct 30 12:10:50.833104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.833040    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"]
Oct 30 12:10:50.926222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.926172    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"]
Oct 30 12:10:50.944837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:50.944808    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:51.630882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.630854    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" event={"ID":"88864962-1518-4fec-82dc-a65b4afe8214","Type":"ContainerStarted","Data":"bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272"}
Oct 30 12:10:51.631041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.630908    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" event={"ID":"88864962-1518-4fec-82dc-a65b4afe8214","Type":"ContainerStarted","Data":"aa026732bd36099bf6dd28c5c78cbb98ab0f2708e4785fb1b340c788c23885ba"}
Oct 30 12:10:51.631041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.631029    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:10:51.634714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.634690    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" event={"ID":"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac","Type":"ContainerStarted","Data":"e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1"}
Oct 30 12:10:51.634841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.634730    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" event={"ID":"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac","Type":"ContainerStarted","Data":"e9d11f05e1576c10e150d6a2d831451915f1cace4ef35b31a175e5283931e88a"}
Oct 30 12:10:51.634841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.634768    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:10:51.671834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.671727    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" podStartSLOduration=1.671702452 podStartE2EDuration="1.671702452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:51.671011499 +0000 UTC m=+2455.465895997" watchObservedRunningTime="2026-10-30 12:10:51.671702452 +0000 UTC m=+2455.466586946"
Oct 30 12:10:51.717076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.717024    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" podStartSLOduration=1.717007722 podStartE2EDuration="1.717007722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:51.712593892 +0000 UTC m=+2455.507478385" watchObservedRunningTime="2026-10-30 12:10:51.717007722 +0000 UTC m=+2455.511892219"
Oct 30 12:10:51.808484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.808374    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:51.871409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.871381    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:51.942573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.942541    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc\" (UniqueName: \"kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc\") pod \"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed\" (UID: \"7aeb5054-03c7-48d1-8b47-59cb2cd2cbed\") "
Oct 30 12:10:51.944879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:51.944853    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc" pod "7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" (UID: "7aeb5054-03c7-48d1-8b47-59cb2cd2cbed"). InnerVolumeSpecName "kube-api-access-744tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:52.044204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.044163    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-744tc\" (UniqueName: \"kubernetes.io/projected/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed-kube-api-access-744tc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:10:52.335889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.335849    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" path="/var/lib/kubelet/pods/7aeb5054-03c7-48d1-8b47-59cb2cd2cbed/volumes"
Oct 30 12:10:52.644774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.644691    5060 generic.go:356] "Generic (PLEG): container finished" podID="7aeb5054-03c7-48d1-8b47-59cb2cd2cbed" containerID="18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f" exitCode=137
Oct 30 12:10:52.644774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.644736    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-e41327"
Oct 30 12:10:52.644989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.644815    5060 scope.go:122] "RemoveContainer" containerID="18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f"
Oct 30 12:10:52.655295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.655274    5060 scope.go:122] "RemoveContainer" containerID="18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f"
Oct 30 12:10:52.655666 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:52.655642    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f\": container with ID starting with 18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f not found: ID does not exist" containerID="18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f"
Oct 30 12:10:52.655734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.655676    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f"} err="failed to get container status \"18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f\": rpc error: code = NotFound desc = could not find container \"18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f\": container with ID starting with 18d76b282ead59e4efe605cca4c3b945c8956178a6aa5b7a366f53192776892f not found: ID does not exist"
Oct 30 12:10:52.740752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:52.740714    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:54.439180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:54.439139    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:56.247521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:56.247484    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:57.994352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:57.994317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:59.537296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.537256    5060 scope.go:122] "RemoveContainer" containerID="9d693202fe9be6099a003d2d8adaad24cd4a427eb2bd9fe2cc966e681fa2b041"
Oct 30 12:10:59.548253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.548226    5060 scope.go:122] "RemoveContainer" containerID="1b0585ce6c7df719e457e0ca6de8e3c687166ec17a109860edd093cf1e471c0b"
Oct 30 12:10:59.560070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.560044    5060 scope.go:122] "RemoveContainer" containerID="2cd48f93180d6dd15a30d8ca2fb82ee88e96d339658bcc6535015f14a2de58fc"
Oct 30 12:10:59.571940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.571915    5060 scope.go:122] "RemoveContainer" containerID="cce28aa5e42b92d6fd2db65fc26de9fa301ae307731c2526adfa92a585428513"
Oct 30 12:10:59.672761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.672728    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:10:59.788403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.788323    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-c7fbc8"]
Oct 30 12:10:59.788632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.788576    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-c7fbc8" podUID="a2f503d0-2921-49a3-918b-5ca8682a4339" containerName="test-curl-7377-c7fbc8" containerID="cri-o://dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78" gracePeriod=2
Oct 30 12:10:59.797881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:10:59.795158    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-c7fbc8"]
Oct 30 12:10:59.803098 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:10:59.802426    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-c7fbc8\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="a2f503d0-2921-49a3-918b-5ca8682a4339" pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:11:01.462497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:01.462428    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:02.025973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.025932    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:11:02.073511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.073372    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg\" (UniqueName: \"kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg\") pod \"a2f503d0-2921-49a3-918b-5ca8682a4339\" (UID: \"a2f503d0-2921-49a3-918b-5ca8682a4339\") "
Oct 30 12:11:02.075765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.075726    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg" pod "a2f503d0-2921-49a3-918b-5ca8682a4339" (UID: "a2f503d0-2921-49a3-918b-5ca8682a4339"). InnerVolumeSpecName "kube-api-access-s6xmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:02.086545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.086503    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:02.174730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.174694    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xmg\" (UniqueName: \"kubernetes.io/projected/a2f503d0-2921-49a3-918b-5ca8682a4339-kube-api-access-s6xmg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:02.336369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.336283    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f503d0-2921-49a3-918b-5ca8682a4339" path="/var/lib/kubelet/pods/a2f503d0-2921-49a3-918b-5ca8682a4339/volumes"
Oct 30 12:11:02.649211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.649130    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:11:02.649211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.649208    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:11:02.734718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.734666    5060 generic.go:356] "Generic (PLEG): container finished" podID="a2f503d0-2921-49a3-918b-5ca8682a4339" containerID="dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78" exitCode=137
Oct 30 12:11:02.734718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.734715    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-c7fbc8"
Oct 30 12:11:02.734944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.734863    5060 scope.go:122] "RemoveContainer" containerID="dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78"
Oct 30 12:11:02.746914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.746884    5060 scope.go:122] "RemoveContainer" containerID="dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78"
Oct 30 12:11:02.747306 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:11:02.747280    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78\": container with ID starting with dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78 not found: ID does not exist" containerID="dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78"
Oct 30 12:11:02.747403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:02.747314    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78"} err="failed to get container status \"dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78\": rpc error: code = NotFound desc = could not find container \"dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78\": container with ID starting with dfbb7f20ac41e73c41561523bcc82a01f76bbbda2216b3bcd182e4439f59cd78 not found: ID does not exist"
Oct 30 12:11:03.244311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:03.244269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:04.925807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:04.925769    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:06.673112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:06.673010    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:08.415717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:08.415676    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:10.187741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:10.187708    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:11.516925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.516884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:11.920339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.920247    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"]
Oct 30 12:11:11.921436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.921408    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a2f503d0-2921-49a3-918b-5ca8682a4339" containerName="test-curl-7377-c7fbc8"
Oct 30 12:11:11.921629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.921614    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f503d0-2921-49a3-918b-5ca8682a4339" containerName="test-curl-7377-c7fbc8"
Oct 30 12:11:11.921921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.921908    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f503d0-2921-49a3-918b-5ca8682a4339" containerName="test-curl-7377-c7fbc8"
Oct 30 12:11:11.926701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.926676    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:11.950735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.950704    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"]
Oct 30 12:11:11.957778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:11.957743    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-4f7443-openshift-default-dockercfg-bsgfq\""
Oct 30 12:11:12.000473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000420    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.000684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000551    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.000684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000604    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.000821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000710    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.000821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.000928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.001011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000930    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.001011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.001011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.000988    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842mq\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.016178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.016148    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:12.102779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.102728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.102995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.102818    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.102995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.102856    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.102995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.102963    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103001    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103040    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103112    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103140    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103197    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842mq\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103235    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103727    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103841    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.103950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.103906    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.106094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.106067    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.106346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.106323    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.127716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.127680    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-842mq\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.127904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.127881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.156811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.156778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:12.245424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.245365    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:12.340475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.331383    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:12.710972 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:12.710942    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b6f25c_bf59_42b1_9d3e_4251827e7a65.slice/crio-083999c0658043b8cd8ea9307306512dba169fdacd89e2dbb9ad305564d4949a WatchSource:0}: Error finding container 083999c0658043b8cd8ea9307306512dba169fdacd89e2dbb9ad305564d4949a: Status 404 returned error can't find the container with id 083999c0658043b8cd8ea9307306512dba169fdacd89e2dbb9ad305564d4949a
Oct 30 12:11:12.712740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.712711    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"]
Oct 30 12:11:12.714658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.714619    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:12.714778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.714707    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:12.714778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.714747    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:12.839475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.837862    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" event={"ID":"77b6f25c-bf59-42b1-9d3e-4251827e7a65","Type":"ContainerStarted","Data":"01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"}
Oct 30 12:11:12.839475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.837949    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" event={"ID":"77b6f25c-bf59-42b1-9d3e-4251827e7a65","Type":"ContainerStarted","Data":"083999c0658043b8cd8ea9307306512dba169fdacd89e2dbb9ad305564d4949a"}
Oct 30 12:11:12.912477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:12.912233    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podStartSLOduration=1.912208884 podStartE2EDuration="1.912208884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:12.891964851 +0000 UTC m=+2476.686849347" watchObservedRunningTime="2026-10-30 12:11:12.912208884 +0000 UTC m=+2476.707093379"
Oct 30 12:11:13.248476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:13.247806    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:13.248476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:13.247945    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 12:11:13.248476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:13.247990    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 12:11:14.251858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:14.251823    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:14.271161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:14.271119    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:14.859424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:14.859391    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:14.860402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:14.860383    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:11:15.149891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.149806    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"]
Oct 30 12:11:15.155581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.155555    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.168313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.168287    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"]
Oct 30 12:11:15.170673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.170649    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:11:15.170673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.170649    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:11:15.170865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.170673    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:11:15.170865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.170648    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:11:15.197299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.196872    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.197299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.196979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.197299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.197064    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8ws\" (UniqueName: \"kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.298526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.298483    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.299026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.298545    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.299026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.298613    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8ws\" (UniqueName: \"kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.299287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.299267    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.301164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.301140    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.322375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.322337    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8ws\" (UniqueName: \"kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws\") pod \"maas-api-revoke-sub-11e18ad2-27b9x\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.429812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.429705    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:15.472535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.472496    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:15.587752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.585588    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:15.787346 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:15.787302    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cbfb11_095e_4a6b_93c4_df330286262d.slice/crio-188dc7f75f6a5c72ec9e5c55e0cc3dfe7c8e71a016e7103e80df738f3995ba1b WatchSource:0}: Error finding container 188dc7f75f6a5c72ec9e5c55e0cc3dfe7c8e71a016e7103e80df738f3995ba1b: Status 404 returned error can't find the container with id 188dc7f75f6a5c72ec9e5c55e0cc3dfe7c8e71a016e7103e80df738f3995ba1b
Oct 30 12:11:15.789186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.789159    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"]
Oct 30 12:11:15.869904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:15.869866    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x" event={"ID":"21cbfb11-095e-4a6b-93c4-df330286262d","Type":"ContainerStarted","Data":"188dc7f75f6a5c72ec9e5c55e0cc3dfe7c8e71a016e7103e80df738f3995ba1b"}
Oct 30 12:11:16.280883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:16.280847    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:16.879138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:16.879104    5060 generic.go:356] "Generic (PLEG): container finished" podID="21cbfb11-095e-4a6b-93c4-df330286262d" containerID="3fa3b23389fd56a782e013b84c7a249d1f33317a3c0cfa0e46c56effc919d327" exitCode=0
Oct 30 12:11:16.879658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:16.879217    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x" event={"ID":"21cbfb11-095e-4a6b-93c4-df330286262d","Type":"ContainerDied","Data":"3fa3b23389fd56a782e013b84c7a249d1f33317a3c0cfa0e46c56effc919d327"}
Oct 30 12:11:17.167476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.167424    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:17.317335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.317299    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"]
Oct 30 12:11:17.322650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.322631    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.330334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.330165    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:17.337718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.337694    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-4f7443\""
Oct 30 12:11:17.337925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.337728    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-4f7443-dockercfg-996zw\""
Oct 30 12:11:17.338284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.338264    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"]
Oct 30 12:11:17.436525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.426031    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"]
Oct 30 12:11:17.436525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.431696    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.443477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.442562    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.443477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.442643    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2s2f\" (UniqueName: \"kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f\") pod \"payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.482479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.478357    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"]
Oct 30 12:11:17.543964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.543931    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.544173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.543994    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmr4m\" (UniqueName: \"kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m\") pod \"payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.544173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.544030    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2s2f\" (UniqueName: \"kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f\") pod \"payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.544298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.544251    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.544934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.544908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.567602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.567575    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2s2f\" (UniqueName: \"kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f\") pod \"payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.639939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.639908    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:17.645746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.645717    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.645867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.645845    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmr4m\" (UniqueName: \"kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m\") pod \"payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.646336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.646316    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.669648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.669616    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmr4m\" (UniqueName: \"kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m\") pod \"payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.827878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.827840    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:17.954996 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:17.954956    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18661ff5_1c6a_4e60_adfe_e036eb1ad34d.slice/crio-3c40a7ebbdf33caa376d09703e2241d068c050eccb6d9472121ba64e00e35b3d WatchSource:0}: Error finding container 3c40a7ebbdf33caa376d09703e2241d068c050eccb6d9472121ba64e00e35b3d: Status 404 returned error can't find the container with id 3c40a7ebbdf33caa376d09703e2241d068c050eccb6d9472121ba64e00e35b3d
Oct 30 12:11:17.958188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:17.958159    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"]
Oct 30 12:11:18.078156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.078129    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:18.141801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.140713    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"]
Oct 30 12:11:18.141801 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:18.140828    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34ea3a0_dffb_41c5_82f1_ba407b590721.slice/crio-83a3ec0dc00d6fa4c79d0ddfc22e70f510ad6e1d4f07e64d15c32c704a152268 WatchSource:0}: Error finding container 83a3ec0dc00d6fa4c79d0ddfc22e70f510ad6e1d4f07e64d15c32c704a152268: Status 404 returned error can't find the container with id 83a3ec0dc00d6fa4c79d0ddfc22e70f510ad6e1d4f07e64d15c32c704a152268
Oct 30 12:11:18.157673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.157642    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca\") pod \"21cbfb11-095e-4a6b-93c4-df330286262d\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") "
Oct 30 12:11:18.157962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.157931    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws\" (UniqueName: \"kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws\") pod \"21cbfb11-095e-4a6b-93c4-df330286262d\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") "
Oct 30 12:11:18.158078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.157979    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca\") pod \"21cbfb11-095e-4a6b-93c4-df330286262d\" (UID: \"21cbfb11-095e-4a6b-93c4-df330286262d\") "
Oct 30 12:11:18.158602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.158573    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca" pod "21cbfb11-095e-4a6b-93c4-df330286262d" (UID: "21cbfb11-095e-4a6b-93c4-df330286262d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:18.160847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.160823    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws" pod "21cbfb11-095e-4a6b-93c4-df330286262d" (UID: "21cbfb11-095e-4a6b-93c4-df330286262d"). InnerVolumeSpecName "kube-api-access-ds8ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:18.160940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.160828    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca" pod "21cbfb11-095e-4a6b-93c4-df330286262d" (UID: "21cbfb11-095e-4a6b-93c4-df330286262d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:11:18.166096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.166055    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:18.259504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.259471    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:18.259504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.259502    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8ws\" (UniqueName: \"kubernetes.io/projected/21cbfb11-095e-4a6b-93c4-df330286262d-kube-api-access-ds8ws\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:18.259741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.259515    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cbfb11-095e-4a6b-93c4-df330286262d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:18.896431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.896343    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"
Oct 30 12:11:18.896431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.896378    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x" event={"ID":"21cbfb11-095e-4a6b-93c4-df330286262d","Type":"ContainerDied","Data":"188dc7f75f6a5c72ec9e5c55e0cc3dfe7c8e71a016e7103e80df738f3995ba1b"}
Oct 30 12:11:18.896431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.896425    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="188dc7f75f6a5c72ec9e5c55e0cc3dfe7c8e71a016e7103e80df738f3995ba1b"
Oct 30 12:11:18.898580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.898547    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" event={"ID":"e34ea3a0-dffb-41c5-82f1-ba407b590721","Type":"ContainerStarted","Data":"935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc"}
Oct 30 12:11:18.898740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.898601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" event={"ID":"e34ea3a0-dffb-41c5-82f1-ba407b590721","Type":"ContainerStarted","Data":"83a3ec0dc00d6fa4c79d0ddfc22e70f510ad6e1d4f07e64d15c32c704a152268"}
Oct 30 12:11:18.899289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.899267    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:18.900659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.900634    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" event={"ID":"18661ff5-1c6a-4e60-adfe-e036eb1ad34d","Type":"ContainerStarted","Data":"057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4"}
Oct 30 12:11:18.900759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.900675    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" event={"ID":"18661ff5-1c6a-4e60-adfe-e036eb1ad34d","Type":"ContainerStarted","Data":"3c40a7ebbdf33caa376d09703e2241d068c050eccb6d9472121ba64e00e35b3d"}
Oct 30 12:11:18.900759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.900703    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:18.945370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:18.945321    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" podStartSLOduration=1.94530775 podStartE2EDuration="1.94530775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:18.943773331 +0000 UTC m=+2482.738657825" watchObservedRunningTime="2026-10-30 12:11:18.94530775 +0000 UTC m=+2482.740192263"
Oct 30 12:11:19.002098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:19.002048    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" podStartSLOduration=2.002033428 podStartE2EDuration="2.002033428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:18.998380456 +0000 UTC m=+2482.793264951" watchObservedRunningTime="2026-10-30 12:11:19.002033428 +0000 UTC m=+2482.796917921"
Oct 30 12:11:19.857809 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:11:19.857772    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:11:19.953377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:19.953332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:21.640371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:21.640331    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:23.331267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:23.331225    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:25.050803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.050763    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:25.500843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.500803    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"]
Oct 30 12:11:25.501829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.501797    5060 cpu_manager.go:414] "containerMap: removing container" podUID="21cbfb11-095e-4a6b-93c4-df330286262d" containerName="revoke-keys"
Oct 30 12:11:25.501829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.501823    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cbfb11-095e-4a6b-93c4-df330286262d" containerName="revoke-keys"
Oct 30 12:11:25.502084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.502024    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cbfb11-095e-4a6b-93c4-df330286262d" containerName="revoke-keys"
Oct 30 12:11:25.506567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.506539    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.528339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.528300    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"]
Oct 30 12:11:25.534828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.534799    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:11:25.535040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.534829    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:11:25.535040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.534829    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:11:25.535040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.534830    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:11:25.660190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.660154    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87g4\" (UniqueName: \"kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.660190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.660197    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.660414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.660354    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.762196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.762102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.762392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.762233    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87g4\" (UniqueName: \"kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.762392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.762255    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.762975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.762949    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.764921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.764897    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.787014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.786975    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87g4\" (UniqueName: \"kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4\") pod \"maas-api-revoke-sub-6b5f53b7-z5wbh\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:25.832045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:25.832014    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:26.139335 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:26.139302    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3e9406_2204_419c_9e15_35dd0c14b285.slice/crio-14e87a135b7320c7b33e2051a3e9c369db046d66ade519c2b6f3cfd3b53e4486 WatchSource:0}: Error finding container 14e87a135b7320c7b33e2051a3e9c369db046d66ade519c2b6f3cfd3b53e4486: Status 404 returned error can't find the container with id 14e87a135b7320c7b33e2051a3e9c369db046d66ade519c2b6f3cfd3b53e4486
Oct 30 12:11:26.141355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.141328    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"]
Oct 30 12:11:26.194933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.194894    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"]
Oct 30 12:11:26.200611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.200583    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.218429    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"]
Oct 30 12:11:26.370123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.370085    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.370356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.370293    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnh9l\" (UniqueName: \"kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.370422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.370383    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.471843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.471800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.472096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.472029    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnh9l\" (UniqueName: \"kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.472648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.472622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.472779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.472630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.475379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.475355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.496140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.496107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnh9l\" (UniqueName: \"kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l\") pod \"maas-api-revoke-sub-0c800233-chmbt\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.522696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.522655    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:26.832361 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:26.832333    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94410c7d_6b0f_4ae2_8141_b86b2c2ce34b.slice/crio-2a23865125c23ed643f03062a5df193635947ca4e8d1eb4261efd4d5b3e7f5b1 WatchSource:0}: Error finding container 2a23865125c23ed643f03062a5df193635947ca4e8d1eb4261efd4d5b3e7f5b1: Status 404 returned error can't find the container with id 2a23865125c23ed643f03062a5df193635947ca4e8d1eb4261efd4d5b3e7f5b1
Oct 30 12:11:26.835010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.834979    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"]
Oct 30 12:11:26.857264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.857230    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:26.967861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.967824    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt" event={"ID":"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b","Type":"ContainerStarted","Data":"2a23865125c23ed643f03062a5df193635947ca4e8d1eb4261efd4d5b3e7f5b1"}
Oct 30 12:11:26.969670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.969644    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh" event={"ID":"4b3e9406-2204-419c-9e15-35dd0c14b285","Type":"ContainerStarted","Data":"b7b25baf7040e7d147c13ce9052d78b4768f65be4a0bbb7d3426b0e946adf01d"}
Oct 30 12:11:26.969837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:26.969695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh" event={"ID":"4b3e9406-2204-419c-9e15-35dd0c14b285","Type":"ContainerStarted","Data":"14e87a135b7320c7b33e2051a3e9c369db046d66ade519c2b6f3cfd3b53e4486"}
Oct 30 12:11:27.012591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:27.012525    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh" podStartSLOduration=2.012507219 podStartE2EDuration="2.012507219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:27.010497365 +0000 UTC m=+2490.805381850" watchObservedRunningTime="2026-10-30 12:11:27.012507219 +0000 UTC m=+2490.807391713"
Oct 30 12:11:27.980258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:27.980222    5060 generic.go:356] "Generic (PLEG): container finished" podID="94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" containerID="930eaad18eada84d389a634f776f17b9e514569fddb1cd9f8edc152e8c47208e" exitCode=0
Oct 30 12:11:27.980862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:27.980324    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt" event={"ID":"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b","Type":"ContainerDied","Data":"930eaad18eada84d389a634f776f17b9e514569fddb1cd9f8edc152e8c47208e"}
Oct 30 12:11:27.982350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:27.982320    5060 generic.go:356] "Generic (PLEG): container finished" podID="4b3e9406-2204-419c-9e15-35dd0c14b285" containerID="b7b25baf7040e7d147c13ce9052d78b4768f65be4a0bbb7d3426b0e946adf01d" exitCode=0
Oct 30 12:11:27.982491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:27.982399    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh" event={"ID":"4b3e9406-2204-419c-9e15-35dd0c14b285","Type":"ContainerDied","Data":"b7b25baf7040e7d147c13ce9052d78b4768f65be4a0bbb7d3426b0e946adf01d"}
Oct 30 12:11:28.652375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:28.651440    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:29.201650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.197896    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:29.237068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.237029    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:29.318228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.318127    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca\") pod \"4b3e9406-2204-419c-9e15-35dd0c14b285\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") "
Oct 30 12:11:29.318427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.318245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca\") pod \"4b3e9406-2204-419c-9e15-35dd0c14b285\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") "
Oct 30 12:11:29.318427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.318349    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4\" (UniqueName: \"kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4\") pod \"4b3e9406-2204-419c-9e15-35dd0c14b285\" (UID: \"4b3e9406-2204-419c-9e15-35dd0c14b285\") "
Oct 30 12:11:29.318578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.318505    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca" pod "4b3e9406-2204-419c-9e15-35dd0c14b285" (UID: "4b3e9406-2204-419c-9e15-35dd0c14b285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:29.319288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.319145    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:29.320615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.320589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca" pod "4b3e9406-2204-419c-9e15-35dd0c14b285" (UID: "4b3e9406-2204-419c-9e15-35dd0c14b285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:11:29.321089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.321071    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4" pod "4b3e9406-2204-419c-9e15-35dd0c14b285" (UID: "4b3e9406-2204-419c-9e15-35dd0c14b285"). InnerVolumeSpecName "kube-api-access-l87g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:29.420018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.419979    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca\") pod \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") "
Oct 30 12:11:29.420259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.420052    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l\" (UniqueName: \"kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l\") pod \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") "
Oct 30 12:11:29.420259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.420092    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca\") pod \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\" (UID: \"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b\") "
Oct 30 12:11:29.420549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.420516    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca" pod "94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" (UID: "94410c7d-6b0f-4ae2-8141-b86b2c2ce34b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:29.420828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.420770    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87g4\" (UniqueName: \"kubernetes.io/projected/4b3e9406-2204-419c-9e15-35dd0c14b285-kube-api-access-l87g4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:29.420828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.420798    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:29.420828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.420814    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b3e9406-2204-419c-9e15-35dd0c14b285-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:29.422486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.422443    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca" pod "94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" (UID: "94410c7d-6b0f-4ae2-8141-b86b2c2ce34b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:11:29.422838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.422806    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l" pod "94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" (UID: "94410c7d-6b0f-4ae2-8141-b86b2c2ce34b"). InnerVolumeSpecName "kube-api-access-qnh9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:29.522425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.522386    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:29.522425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.522426    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnh9l\" (UniqueName: \"kubernetes.io/projected/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b-kube-api-access-qnh9l\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:29.911948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.911918    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:11:29.912152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:29.911989    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:11:30.009426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.009395    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"
Oct 30 12:11:30.010466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.009437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt" event={"ID":"94410c7d-6b0f-4ae2-8141-b86b2c2ce34b","Type":"ContainerDied","Data":"2a23865125c23ed643f03062a5df193635947ca4e8d1eb4261efd4d5b3e7f5b1"}
Oct 30 12:11:30.010717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.010689    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a23865125c23ed643f03062a5df193635947ca4e8d1eb4261efd4d5b3e7f5b1"
Oct 30 12:11:30.012747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.012718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh" event={"ID":"4b3e9406-2204-419c-9e15-35dd0c14b285","Type":"ContainerDied","Data":"14e87a135b7320c7b33e2051a3e9c369db046d66ade519c2b6f3cfd3b53e4486"}
Oct 30 12:11:30.012931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.012914    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14e87a135b7320c7b33e2051a3e9c369db046d66ade519c2b6f3cfd3b53e4486"
Oct 30 12:11:30.013103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.013081    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"
Oct 30 12:11:30.433901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:30.433796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:32.246584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:32.246548    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:34.003009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:34.002975    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:35.697489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:35.693167    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:36.036638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.036541    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"]
Oct 30 12:11:36.048900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.048861    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf6585a-c977j"]
Oct 30 12:11:36.337394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.337318    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b3a431-f821-41e4-a4bd-32b463be5b23" path="/var/lib/kubelet/pods/78b3a431-f821-41e4-a4bd-32b463be5b23/volumes"
Oct 30 12:11:36.556036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.555996    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"]
Oct 30 12:11:36.556652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.556637    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4b3e9406-2204-419c-9e15-35dd0c14b285" containerName="revoke-keys"
Oct 30 12:11:36.556727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.556653    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3e9406-2204-419c-9e15-35dd0c14b285" containerName="revoke-keys"
Oct 30 12:11:36.556727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.556662    5060 cpu_manager.go:414] "containerMap: removing container" podUID="94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" containerName="revoke-keys"
Oct 30 12:11:36.556727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.556667    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" containerName="revoke-keys"
Oct 30 12:11:36.556829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.556793    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3e9406-2204-419c-9e15-35dd0c14b285" containerName="revoke-keys"
Oct 30 12:11:36.556829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.556800    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" containerName="revoke-keys"
Oct 30 12:11:36.561497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.561477    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.603531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.594814    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:11:36.603531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.600649    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:11:36.603531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.600736    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:11:36.608118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.608083    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"]
Oct 30 12:11:36.644948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.644742    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.644948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.644896    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppvk\" (UniqueName: \"kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.645263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.645151    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.708480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.707705    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:11:36.746351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.746315    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.746601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.746390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppvk\" (UniqueName: \"kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.746601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.746582    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.747376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.747350    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.748828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.748807    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.770653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.770620    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppvk\" (UniqueName: \"kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk\") pod \"maas-api-revoke-sub-e06322cf-rxcgk\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:36.875715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:36.875627    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:37.208156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:37.208121    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"]
Oct 30 12:11:37.208775 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:37.208746    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeec1a4e_7269_4d70_a5ba_abf40f031a0b.slice/crio-fbac6108d5dcb1881e436ab36bf18b81ac6f7ecaf973ec09298117ff97e943d6 WatchSource:0}: Error finding container fbac6108d5dcb1881e436ab36bf18b81ac6f7ecaf973ec09298117ff97e943d6: Status 404 returned error can't find the container with id fbac6108d5dcb1881e436ab36bf18b81ac6f7ecaf973ec09298117ff97e943d6
Oct 30 12:11:37.325780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:37.325743    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"]
Oct 30 12:11:37.325972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:37.325952    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy" containerID="cri-o://f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075" gracePeriod=30
Oct 30 12:11:37.556130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:37.556092    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:37.916936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:37.916840    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"]
Oct 30 12:11:37.975962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:37.975912    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:38.113474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:38.110622    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk" event={"ID":"beec1a4e-7269-4d70-a5ba-abf40f031a0b","Type":"ContainerStarted","Data":"0db2e5e203d563e01c6bd3ba3add0cb73e011adbdb8aeb82e007ea8baba8f199"}
Oct 30 12:11:38.113474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:38.110720    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk" event={"ID":"beec1a4e-7269-4d70-a5ba-abf40f031a0b","Type":"ContainerStarted","Data":"fbac6108d5dcb1881e436ab36bf18b81ac6f7ecaf973ec09298117ff97e943d6"}
Oct 30 12:11:38.157543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:38.157476    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk" podStartSLOduration=2.1574372 podStartE2EDuration="2.1574372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:38.15669619 +0000 UTC m=+2501.951580683" watchObservedRunningTime="2026-10-30 12:11:38.1574372 +0000 UTC m=+2501.952321694"
Oct 30 12:11:39.122379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:39.122340    5060 generic.go:356] "Generic (PLEG): container finished" podID="beec1a4e-7269-4d70-a5ba-abf40f031a0b" containerID="0db2e5e203d563e01c6bd3ba3add0cb73e011adbdb8aeb82e007ea8baba8f199" exitCode=0
Oct 30 12:11:39.122874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:39.122426    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk" event={"ID":"beec1a4e-7269-4d70-a5ba-abf40f031a0b","Type":"ContainerDied","Data":"0db2e5e203d563e01c6bd3ba3add0cb73e011adbdb8aeb82e007ea8baba8f199"}
Oct 30 12:11:39.372672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:39.372569    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:40.326847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.326818    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:40.409228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.409126    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk\" (UniqueName: \"kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk\") pod \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") "
Oct 30 12:11:40.410477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.409500    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca\") pod \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") "
Oct 30 12:11:40.410477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.409626    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca\") pod \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\" (UID: \"beec1a4e-7269-4d70-a5ba-abf40f031a0b\") "
Oct 30 12:11:40.410477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.409902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca" pod "beec1a4e-7269-4d70-a5ba-abf40f031a0b" (UID: "beec1a4e-7269-4d70-a5ba-abf40f031a0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:40.410477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.410284    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:40.413150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.413121    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca" pod "beec1a4e-7269-4d70-a5ba-abf40f031a0b" (UID: "beec1a4e-7269-4d70-a5ba-abf40f031a0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:11:40.413612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.413587    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk" pod "beec1a4e-7269-4d70-a5ba-abf40f031a0b" (UID: "beec1a4e-7269-4d70-a5ba-abf40f031a0b"). InnerVolumeSpecName "kube-api-access-2ppvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:40.511313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.511274    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beec1a4e-7269-4d70-a5ba-abf40f031a0b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:40.511313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:40.511312    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppvk\" (UniqueName: \"kubernetes.io/projected/beec1a4e-7269-4d70-a5ba-abf40f031a0b-kube-api-access-2ppvk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:41.071724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:41.071682    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:41.141634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:41.141600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"
Oct 30 12:11:41.141837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:41.141632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk" event={"ID":"beec1a4e-7269-4d70-a5ba-abf40f031a0b","Type":"ContainerDied","Data":"fbac6108d5dcb1881e436ab36bf18b81ac6f7ecaf973ec09298117ff97e943d6"}
Oct 30 12:11:41.141837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:41.141683    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbac6108d5dcb1881e436ab36bf18b81ac6f7ecaf973ec09298117ff97e943d6"
Oct 30 12:11:41.233231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:41.223391    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:41.364584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:41.364486    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:42.175475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:42.174682    5060 generic.go:356] "Generic (PLEG): container finished" podID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerID="7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6" exitCode=0
Oct 30 12:11:42.175475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:42.174736    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" event={"ID":"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9","Type":"ContainerDied","Data":"7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"}
Oct 30 12:11:42.175475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:42.175290    5060 scope.go:122] "RemoveContainer" containerID="7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"
Oct 30 12:11:42.178560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:42.178357    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:42.178560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:42.178475    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:42.178560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:42.178526    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:43.034963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.032148    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:11:43.048501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.045414    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:43.162854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.162761    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.162854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.162843    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.162865    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.162905    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.162941    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.162976    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163061    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163107    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:43.163547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163169    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:43.163547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163213    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy\") pod \"9bae7157-5ca6-42fc-9a4c-659843476660\" (UID: \"9bae7157-5ca6-42fc-9a4c-659843476660\") "
Oct 30 12:11:43.163547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163300    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:43.163715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163574    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:43.163715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163606    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:43.163840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163821    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.163912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163845    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.163912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163859    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.163912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163874    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bae7157-5ca6-42fc-9a4c-659843476660-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.163912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.163888    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.165639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.165600    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "kube-api-access-gnz65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:43.165639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.165610    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:43.166095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.166074    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:11:43.166178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.166118    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token" pod "9bae7157-5ca6-42fc-9a4c-659843476660" (UID: "9bae7157-5ca6-42fc-9a4c-659843476660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:43.233596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.233552    5060 generic.go:356] "Generic (PLEG): container finished" podID="9bae7157-5ca6-42fc-9a4c-659843476660" containerID="f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075" exitCode=0
Oct 30 12:11:43.233596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.233596    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"
Oct 30 12:11:43.233861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.233627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" event={"ID":"9bae7157-5ca6-42fc-9a4c-659843476660","Type":"ContainerDied","Data":"f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075"}
Oct 30 12:11:43.233861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.233690    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f" event={"ID":"9bae7157-5ca6-42fc-9a4c-659843476660","Type":"ContainerDied","Data":"7898d57ebec26107b00f1a0d8c7c36f0ea301f02b2f3813f81315447988f3126"}
Oct 30 12:11:43.233861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.233716    5060 scope.go:122] "RemoveContainer" containerID="f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075"
Oct 30 12:11:43.236131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.236106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" event={"ID":"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9","Type":"ContainerStarted","Data":"6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a"}
Oct 30 12:11:43.248880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.248856    5060 scope.go:122] "RemoveContainer" containerID="f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075"
Oct 30 12:11:43.249235 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:11:43.249210    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075\": container with ID starting with f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075 not found: ID does not exist" containerID="f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075"
Oct 30 12:11:43.249315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.249242    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075"} err="failed to get container status \"f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075\": rpc error: code = NotFound desc = could not find container \"f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075\": container with ID starting with f73c7d9ca1583d386123a56bcde1556d722779ad21c04fc0918ee63b4f2b7075 not found: ID does not exist"
Oct 30 12:11:43.264808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.264784    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.264808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.264808    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnz65\" (UniqueName: \"kubernetes.io/projected/9bae7157-5ca6-42fc-9a4c-659843476660-kube-api-access-gnz65\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.264808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.264818    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bae7157-5ca6-42fc-9a4c-659843476660-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.265025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.264828    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bae7157-5ca6-42fc-9a4c-659843476660-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.347340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.347306    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"]
Oct 30 12:11:43.353086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.353055    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-rv92f"]
Oct 30 12:11:43.861664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:43.861631    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:44.205478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:44.200893    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:11:44.206938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:44.206587    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 12:11:44.206938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:44.206647    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 12:11:44.339732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:44.339696    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" path="/var/lib/kubelet/pods/9bae7157-5ca6-42fc-9a4c-659843476660/volumes"
Oct 30 12:11:44.860956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:44.860925    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:45.203086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:45.203034    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 12:11:45.203392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:45.203347    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 12:11:46.204479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:46.200860    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 12:11:46.204479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:46.200936    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 12:11:46.572341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:46.572248    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:11:46.644589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:46.644546    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:47.203948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:47.203907    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 12:11:47.204275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:47.204247    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 12:11:48.206491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:48.206030    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 12:11:48.206491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:48.206095    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 12:11:48.507266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:48.507175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:49.207202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:49.207171    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:11:49.208997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:49.208967    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:11:49.858523 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:11:49.858482    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:11:49.999167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:49.999132    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:50.184006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:50.183904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:51.833277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:51.833232    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:51.973996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:51.973962    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:52.944028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:52.943981    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8" podUID="5362b1a9-e415-4e66-a911-5fbe5d698490" containerName="main" containerID="cri-o://a5abb8a9930ea7ba0d8e65b744176cacad9530a7c71fe2d6aa41940659ccac9b" gracePeriod=45
Oct 30 12:11:53.356273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.356228    5060 generic.go:356] "Generic (PLEG): container finished" podID="5362b1a9-e415-4e66-a911-5fbe5d698490" containerID="a5abb8a9930ea7ba0d8e65b744176cacad9530a7c71fe2d6aa41940659ccac9b" exitCode=0
Oct 30 12:11:53.356518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.356342    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8" event={"ID":"5362b1a9-e415-4e66-a911-5fbe5d698490","Type":"ContainerDied","Data":"a5abb8a9930ea7ba0d8e65b744176cacad9530a7c71fe2d6aa41940659ccac9b"}
Oct 30 12:11:53.401628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.401590    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:11:53.471483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.470610    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm\") pod \"5362b1a9-e415-4e66-a911-5fbe5d698490\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") "
Oct 30 12:11:53.471483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.470690    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs\" (UniqueName: \"kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs\") pod \"5362b1a9-e415-4e66-a911-5fbe5d698490\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") "
Oct 30 12:11:53.471483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.470724    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z\" (UniqueName: \"kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z\") pod \"5362b1a9-e415-4e66-a911-5fbe5d698490\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") "
Oct 30 12:11:53.471483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.470899    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache\") pod \"5362b1a9-e415-4e66-a911-5fbe5d698490\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") "
Oct 30 12:11:53.471483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.471010    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir\") pod \"5362b1a9-e415-4e66-a911-5fbe5d698490\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") "
Oct 30 12:11:53.478774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.474427    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache" pod "5362b1a9-e415-4e66-a911-5fbe5d698490" (UID: "5362b1a9-e415-4e66-a911-5fbe5d698490"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:53.478774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.475777    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home\") pod \"5362b1a9-e415-4e66-a911-5fbe5d698490\" (UID: \"5362b1a9-e415-4e66-a911-5fbe5d698490\") "
Oct 30 12:11:53.478774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.476522    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:53.478774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.478516    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir" pod "5362b1a9-e415-4e66-a911-5fbe5d698490" (UID: "5362b1a9-e415-4e66-a911-5fbe5d698490"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:53.478774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.478616    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home" pod "5362b1a9-e415-4e66-a911-5fbe5d698490" (UID: "5362b1a9-e415-4e66-a911-5fbe5d698490"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:53.492364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.492285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z" pod "5362b1a9-e415-4e66-a911-5fbe5d698490" (UID: "5362b1a9-e415-4e66-a911-5fbe5d698490"). InnerVolumeSpecName "kube-api-access-vts7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:53.530809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.519799    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs" pod "5362b1a9-e415-4e66-a911-5fbe5d698490" (UID: "5362b1a9-e415-4e66-a911-5fbe5d698490"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:11:53.534926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.534887    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm" pod "5362b1a9-e415-4e66-a911-5fbe5d698490" (UID: "5362b1a9-e415-4e66-a911-5fbe5d698490"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:53.577777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.577738    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:53.577777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.577770    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:53.577777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.577778    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5362b1a9-e415-4e66-a911-5fbe5d698490-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:53.577777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.577786    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5362b1a9-e415-4e66-a911-5fbe5d698490-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:53.578120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.577794    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vts7z\" (UniqueName: \"kubernetes.io/projected/5362b1a9-e415-4e66-a911-5fbe5d698490-kube-api-access-vts7z\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:11:53.645689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:53.645648    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:54.366925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:54.366895    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"
Oct 30 12:11:54.366925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:54.366917    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8" event={"ID":"5362b1a9-e415-4e66-a911-5fbe5d698490","Type":"ContainerDied","Data":"33b9249cf1933ade75271a2e1b48f039ccb728e3e5cb59de38da4390981b152b"}
Oct 30 12:11:54.367557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:54.366977    5060 scope.go:122] "RemoveContainer" containerID="a5abb8a9930ea7ba0d8e65b744176cacad9530a7c71fe2d6aa41940659ccac9b"
Oct 30 12:11:54.425044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:54.425007    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"]
Oct 30 12:11:54.435360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:54.435323    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-28ca05/auth-test-model-28ca05-kserve-64ffcbfcfc-vklk8"]
Oct 30 12:11:55.288903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.288852    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:55.432162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.432128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:55.675004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.674915    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"]
Oct 30 12:11:55.675883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.675858    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy"
Oct 30 12:11:55.675883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.675883    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy"
Oct 30 12:11:55.676091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.675908    5060 cpu_manager.go:414] "containerMap: removing container" podUID="beec1a4e-7269-4d70-a5ba-abf40f031a0b" containerName="revoke-keys"
Oct 30 12:11:55.676091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.675916    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beec1a4e-7269-4d70-a5ba-abf40f031a0b" containerName="revoke-keys"
Oct 30 12:11:55.676091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.675943    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5362b1a9-e415-4e66-a911-5fbe5d698490" containerName="main"
Oct 30 12:11:55.676091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.675951    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5362b1a9-e415-4e66-a911-5fbe5d698490" containerName="main"
Oct 30 12:11:55.676317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.676137    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bae7157-5ca6-42fc-9a4c-659843476660" containerName="istio-proxy"
Oct 30 12:11:55.676317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.676155    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="beec1a4e-7269-4d70-a5ba-abf40f031a0b" containerName="revoke-keys"
Oct 30 12:11:55.676317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.676165    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5362b1a9-e415-4e66-a911-5fbe5d698490" containerName="main"
Oct 30 12:11:55.681768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.681744    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.707160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.707120    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"]
Oct 30 12:11:55.711947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.711922    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2bc4e2-openshift-default-dockercfg-8cmmz\""
Oct 30 12:11:55.761019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.760976    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:55.829528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.807304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.829943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.829910    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.831367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.830186    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.841474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.837770    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.861638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.861591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.862028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.861982    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.862268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.862249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5wj\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.908995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.902713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.950325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.950174    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:55.961770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:55.956028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:56.065778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.065732    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.066445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.066424    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070061    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070171    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070202    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070274    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5wj\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070333    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.070864    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.071084    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.071289    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.072506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.071516    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.073247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.072905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.073932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.073884    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.075663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.074363    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.096275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.096243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.099242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.099216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5wj\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj\") pod \"e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") " pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.114421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.114297    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:56.319116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.319015    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:56.339407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.339371    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5362b1a9-e415-4e66-a911-5fbe5d698490" path="/var/lib/kubelet/pods/5362b1a9-e415-4e66-a911-5fbe5d698490/volumes"
Oct 30 12:11:56.666696 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:56.666656    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892354bd_b329_4d9f_8b3e_a2eda63c55af.slice/crio-979f95adb896967391e2e13a0d20de1fd014a606fd4ca3167d2dde40b8c392a9 WatchSource:0}: Error finding container 979f95adb896967391e2e13a0d20de1fd014a606fd4ca3167d2dde40b8c392a9: Status 404 returned error can't find the container with id 979f95adb896967391e2e13a0d20de1fd014a606fd4ca3167d2dde40b8c392a9
Oct 30 12:11:56.667033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.666931    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"]
Oct 30 12:11:56.669444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.669410    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:56.669589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.669511    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:56.669589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:56.669546    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:57.187729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.187681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:57.423635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.423601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" event={"ID":"892354bd-b329-4d9f-8b3e-a2eda63c55af","Type":"ContainerStarted","Data":"3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"}
Oct 30 12:11:57.423862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.423665    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" event={"ID":"892354bd-b329-4d9f-8b3e-a2eda63c55af","Type":"ContainerStarted","Data":"979f95adb896967391e2e13a0d20de1fd014a606fd4ca3167d2dde40b8c392a9"}
Oct 30 12:11:57.448807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.448771    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:57.499489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.499413    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podStartSLOduration=2.499397427 podStartE2EDuration="2.499397427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:57.485562259 +0000 UTC m=+2521.280446756" watchObservedRunningTime="2026-10-30 12:11:57.499397427 +0000 UTC m=+2521.294281921"
Oct 30 12:11:57.900334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.899676    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"]
Oct 30 12:11:57.905952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.905508    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.928896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.928627    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"]
Oct 30 12:11:57.934598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.934567    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-737f64-openshift-default-dockercfg-4256n\""
Oct 30 12:11:57.988475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.988283    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:57.998118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998080    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998188    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998221    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998274    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5z4\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998505    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:57.998649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:57.998538    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102229    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102350    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102384    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102410    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5z4\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102514    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102691    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.102725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.103632    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.103921    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.104502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.104487    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.150533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.140689    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.150786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.148873    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.168484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.155719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.168484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.155969    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.197478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.171051    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:58.235401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.225095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.235401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.235259    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5z4\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.236032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.236012    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:11:58.324578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.320053    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:58.344483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.342172    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:58.432360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.432276    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:58.433257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.433233    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:11:58.602030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.602000    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"]
Oct 30 12:11:58.604889 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:11:58.604855    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5689375b_457e_44eb_b779_75c5dd8d35fc.slice/crio-4910ec919c2c7a655f39438ba1c70027c9b0f542e19128865852bdc83359b446 WatchSource:0}: Error finding container 4910ec919c2c7a655f39438ba1c70027c9b0f542e19128865852bdc83359b446: Status 404 returned error can't find the container with id 4910ec919c2c7a655f39438ba1c70027c9b0f542e19128865852bdc83359b446
Oct 30 12:11:58.608332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.608293    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:58.608482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.608382    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:58.608482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.608429    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:11:58.971696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:58.971657    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:11:59.441899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.441808    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" event={"ID":"5689375b-457e-44eb-b779-75c5dd8d35fc","Type":"ContainerStarted","Data":"0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"}
Oct 30 12:11:59.441899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.441884    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" event={"ID":"5689375b-457e-44eb-b779-75c5dd8d35fc","Type":"ContainerStarted","Data":"4910ec919c2c7a655f39438ba1c70027c9b0f542e19128865852bdc83359b446"}
Oct 30 12:11:59.490794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.490725    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podStartSLOduration=2.490703639 podStartE2EDuration="2.490703639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:59.486664691 +0000 UTC m=+2523.281549186" watchObservedRunningTime="2026-10-30 12:11:59.490703639 +0000 UTC m=+2523.285588133"
Oct 30 12:11:59.729846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.729804    5060 scope.go:122] "RemoveContainer" containerID="4f693454f7f0b5be44e6547acef8e65cb06f31e3798c430783d400c8cbfb5684"
Oct 30 12:11:59.743333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.743308    5060 scope.go:122] "RemoveContainer" containerID="5eefcc6934746ef0907a8c28191e1155ba0d69473a3eed484175711d7700a644"
Oct 30 12:11:59.755630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.755599    5060 scope.go:122] "RemoveContainer" containerID="d456c8190d71de37bc0e5be5a47bea9b506e2a8c17e25bbbf844912101d5c312"
Oct 30 12:11:59.766910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:11:59.766881    5060 scope.go:122] "RemoveContainer" containerID="ee4a6f31aaf1b5dac17c541527d9c31107b24cbc0b2eb46a5b68f2a27ba645ba"
Oct 30 12:12:00.236800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:00.236769    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:12:00.242774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:00.242744    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:12:00.449238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:00.449200    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:12:00.450257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:00.450235    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:12:00.765325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:00.765290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:01.881218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:01.881176    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:02.029044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:02.029005    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:02.768637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:02.768602    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:03.787255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:03.787220    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:03.922311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:03.922274    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:04.679705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:04.679668    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:06.471050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:06.471004    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:08.144771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:08.144731    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:08.297041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:08.297004    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:08.446209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:08.445666    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:08.590205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:08.590171    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:09.951875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:09.951838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:10.104618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.104565    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:10.341605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.341522    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"]
Oct 30 12:12:10.349164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.349136    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.361843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.361811    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"]
Oct 30 12:12:10.364705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.364675    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2bc4e2-dockercfg-sxwhd\""
Oct 30 12:12:10.364840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.364804    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2bc4e2\""
Oct 30 12:12:10.366427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.366354    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:10.375182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.375160    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.375440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.375418    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mjh\" (UniqueName: \"kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh\") pod \"payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.476831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.476793    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mjh\" (UniqueName: \"kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh\") pod \"payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.477048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.476976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.477878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.477853    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.500701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.500677    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mjh\" (UniqueName: \"kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh\") pod \"payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.609253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.609175    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-ca6130"]
Oct 30 12:12:10.613727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.613708    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:12:10.627940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.627912    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-ca6130"]
Oct 30 12:12:10.628692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.628667    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\""
Oct 30 12:12:10.674808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.674778    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:10.679327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.679302    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4kd\" (UniqueName: \"kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd\") pod \"test-curl-7380-ca6130\" (UID: \"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a\") " pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:12:10.781061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.780594    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4kd\" (UniqueName: \"kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd\") pod \"test-curl-7380-ca6130\" (UID: \"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a\") " pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:12:10.804983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.804954    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4kd\" (UniqueName: \"kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd\") pod \"test-curl-7380-ca6130\" (UID: \"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a\") " pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:12:10.928530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.928438    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:12:10.985618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:10.985587    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"]
Oct 30 12:12:10.986017 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:12:10.985644    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538d1600_10e2_451c_9fdb_67beaaa2335d.slice/crio-c276cf86c971064478eaaad1916262220f379c4fd8bebd0fd515250b1a64bedd WatchSource:0}: Error finding container c276cf86c971064478eaaad1916262220f379c4fd8bebd0fd515250b1a64bedd: Status 404 returned error can't find the container with id c276cf86c971064478eaaad1916262220f379c4fd8bebd0fd515250b1a64bedd
Oct 30 12:12:11.233336 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:12:11.233298    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf28ac3b0_8281_45a1_bc7c_cdeafe72c93a.slice/crio-7c42d4057f67e6af3c777c6ac14e572cb7ce73f14523235b331e54822af101b8 WatchSource:0}: Error finding container 7c42d4057f67e6af3c777c6ac14e572cb7ce73f14523235b331e54822af101b8: Status 404 returned error can't find the container with id 7c42d4057f67e6af3c777c6ac14e572cb7ce73f14523235b331e54822af101b8
Oct 30 12:12:11.234885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.234860    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-ca6130"]
Oct 30 12:12:11.562564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.562437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" event={"ID":"538d1600-10e2-451c-9fdb-67beaaa2335d","Type":"ContainerStarted","Data":"40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a"}
Oct 30 12:12:11.562564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.562522    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:11.562564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.562540    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" event={"ID":"538d1600-10e2-451c-9fdb-67beaaa2335d","Type":"ContainerStarted","Data":"c276cf86c971064478eaaad1916262220f379c4fd8bebd0fd515250b1a64bedd"}
Oct 30 12:12:11.564126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.564104    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-ca6130" event={"ID":"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a","Type":"ContainerStarted","Data":"75c20a08c7575b39a558caa82e91422257cc677aa8bd182e6a5683075d656a1b"}
Oct 30 12:12:11.564256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.564132    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-ca6130" event={"ID":"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a","Type":"ContainerStarted","Data":"7c42d4057f67e6af3c777c6ac14e572cb7ce73f14523235b331e54822af101b8"}
Oct 30 12:12:11.607310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.607239    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" podStartSLOduration=1.6072182000000002 podStartE2EDuration="1.6072182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:11.601930767 +0000 UTC m=+2535.396815273" watchObservedRunningTime="2026-10-30 12:12:11.6072182 +0000 UTC m=+2535.402102695"
Oct 30 12:12:11.639928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:11.639860    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7380-ca6130" podStartSLOduration=1.639839996 podStartE2EDuration="1.639839996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:11.638858265 +0000 UTC m=+2535.433742761" watchObservedRunningTime="2026-10-30 12:12:11.639839996 +0000 UTC m=+2535.434724495"
Oct 30 12:12:12.149513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:12.149443    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:13.613163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:13.613115    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"]
Oct 30 12:12:13.613746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:13.613370    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" containerID="cri-o://23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6" gracePeriod=30
Oct 30 12:12:14.005100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:14.005060    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:15.676512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:15.676480    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:16.032766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:16.032724    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:12:16.032964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:16.032802    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:12:17.459874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:17.459836    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:18.286336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:18.286286    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:18.902402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:18.902366    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 12:12:18.973069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:18.972431    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:19.014652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014618    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.014860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014689    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.014860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014769    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.014860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014822    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.015065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.015065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.015065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014921    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:19.015065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.014988    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.015065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015022    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.015348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015061    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:19.015348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015164    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo\") pod \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\" (UID: \"de97e275-4658-4d9e-b5a9-ccf8c52321d4\") "
Oct 30 12:12:19.015497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015360    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:19.015497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015417    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:12:19.015497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015429    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:19.015968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015919    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.015968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015947    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.015968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015964    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.016211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015979    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.016211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.015992    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.017478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.017415    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "kube-api-access-49nt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:12:19.017595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.017561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:12:19.017888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.017861    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:19.018023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.018003    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo" pod "de97e275-4658-4d9e-b5a9-ccf8c52321d4" (UID: "de97e275-4658-4d9e-b5a9-ccf8c52321d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:12:19.117906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.117870    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.117906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.117909    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nt4\" (UniqueName: \"kubernetes.io/projected/de97e275-4658-4d9e-b5a9-ccf8c52321d4-kube-api-access-49nt4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.118131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.117924    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.118131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.117939    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de97e275-4658-4d9e-b5a9-ccf8c52321d4-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:12:19.257426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.257373    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:19.466020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.465979    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:19.652527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.651520    5060 generic.go:356] "Generic (PLEG): container finished" podID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerID="23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6" exitCode=0
Oct 30 12:12:19.652527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.651691    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"
Oct 30 12:12:19.652527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.651712    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" event={"ID":"de97e275-4658-4d9e-b5a9-ccf8c52321d4","Type":"ContainerDied","Data":"23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6"}
Oct 30 12:12:19.652527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.651762    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c" event={"ID":"de97e275-4658-4d9e-b5a9-ccf8c52321d4","Type":"ContainerDied","Data":"b8da070b8f11644bee58bd4d1ef5a37939a0bdb323847e63da511fd4c8a6d1b0"}
Oct 30 12:12:19.652527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.651788    5060 scope.go:122] "RemoveContainer" containerID="23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6"
Oct 30 12:12:19.668703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.668675    5060 scope.go:122] "RemoveContainer" containerID="23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6"
Oct 30 12:12:19.669103 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:12:19.669059    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6\": container with ID starting with 23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6 not found: ID does not exist" containerID="23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6"
Oct 30 12:12:19.669244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.669094    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6"} err="failed to get container status \"23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6\": rpc error: code = NotFound desc = could not find container \"23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6\": container with ID starting with 23b58ad53cfb93cb67a8929edcff0ae89c52c0ff82b169577217fd551ad80eb6 not found: ID does not exist"
Oct 30 12:12:19.708573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.708526    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"]
Oct 30 12:12:19.712654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.712625    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-r2h5c"]
Oct 30 12:12:19.769263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:19.769227    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:19.859029 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:12:19.858991    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:12:20.339080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:20.339047    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" path="/var/lib/kubelet/pods/de97e275-4658-4d9e-b5a9-ccf8c52321d4/volumes"
Oct 30 12:12:21.029854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:21.029811    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:22.575196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:22.575168    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:12:22.979978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:22.979947    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:24.760658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:24.760615    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:26.357966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:26.357930    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:26.525297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:26.525260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:28.275227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:28.275181    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:30.079324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:30.079291    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:30.223386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:30.223350    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:30.362129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:30.362042    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:30.901632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:30.901587    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:12:30.901826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:30.901645    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:12:31.959087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:31.959053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:32.806936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:32.806904    5060 generic.go:356] "Generic (PLEG): container finished" podID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerID="01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe" exitCode=0
Oct 30 12:12:32.807554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:32.806972    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" event={"ID":"77b6f25c-bf59-42b1-9d3e-4251827e7a65","Type":"ContainerDied","Data":"01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"}
Oct 30 12:12:32.807554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:32.807469    5060 scope.go:122] "RemoveContainer" containerID="01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"
Oct 30 12:12:32.809588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:32.809551    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:12:32.809708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:32.809643    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:12:32.809708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:32.809685    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:12:33.159274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:33.159179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:33.304265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:33.304217    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:33.833708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:33.833665    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" event={"ID":"77b6f25c-bf59-42b1-9d3e-4251827e7a65","Type":"ContainerStarted","Data":"b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e"}
Oct 30 12:12:33.973240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:33.973202    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:34.245826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:34.245789    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:12:34.246229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:34.245987    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 12:12:34.246229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:34.246028    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 12:12:34.875492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:34.871642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:35.019685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.018168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:35.245634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.232548    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"]
Oct 30 12:12:35.245634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.233541    5060 cpu_manager.go:414] "containerMap: removing container" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy"
Oct 30 12:12:35.245634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.233562    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy"
Oct 30 12:12:35.245634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.233794    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de97e275-4658-4d9e-b5a9-ccf8c52321d4" containerName="istio-proxy"
Oct 30 12:12:35.293480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.290616    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.324419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.323567    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 12:12:35.324419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.323632    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 12:12:35.331243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.331216    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"]
Oct 30 12:12:35.331243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.331234    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3e23ec\""
Oct 30 12:12:35.331488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.331404    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3e23ec-dockercfg-fcj6x\""
Oct 30 12:12:35.410017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.397927    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpjd\" (UniqueName: \"kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd\") pod \"payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") " pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.410017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.409736    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume\") pod \"payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") " pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.511316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.511223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpjd\" (UniqueName: \"kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd\") pod \"payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") " pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.512412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.512390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume\") pod \"payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") " pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.513657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.513375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume\") pod \"payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") " pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.535656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.535632    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpjd\" (UniqueName: \"kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd\") pod \"payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") " pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.641443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.641409    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:35.840762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:35.840610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:36.016508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.015347    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"]
Oct 30 12:12:36.016959 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:12:36.016928    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd3967e_2f19_49d2_b38c_bc642f332ddf.slice/crio-8c1079f2e038b48781647da859f7b6745a78a593b5dd5533f9d62f5b576f0b68 WatchSource:0}: Error finding container 8c1079f2e038b48781647da859f7b6745a78a593b5dd5533f9d62f5b576f0b68: Status 404 returned error can't find the container with id 8c1079f2e038b48781647da859f7b6745a78a593b5dd5533f9d62f5b576f0b68
Oct 30 12:12:36.246410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.246326    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 12:12:36.246410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.246389    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 12:12:36.868703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.868660    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6" event={"ID":"ebd3967e-2f19-49d2-b38c-bc642f332ddf","Type":"ContainerStarted","Data":"e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8"}
Oct 30 12:12:36.868890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.868725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6" event={"ID":"ebd3967e-2f19-49d2-b38c-bc642f332ddf","Type":"ContainerStarted","Data":"8c1079f2e038b48781647da859f7b6745a78a593b5dd5533f9d62f5b576f0b68"}
Oct 30 12:12:36.868890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.868754    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:36.925862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:36.925802    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6" podStartSLOduration=1.9257866940000001 podStartE2EDuration="1.925786694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:36.921897228 +0000 UTC m=+2560.716781722" watchObservedRunningTime="2026-10-30 12:12:36.925786694 +0000 UTC m=+2560.720671238"
Oct 30 12:12:37.246331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:37.246291    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 12:12:37.246552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:37.246342    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 12:12:37.637674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:37.637594    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:37.773346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:37.773313    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:38.245798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:38.245764    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 12:12:38.245988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:38.245815    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 12:12:39.250471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:39.250425    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:12:39.250954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:39.250901    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:12:39.251753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:39.251730    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:12:39.423510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:39.423442    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:39.570222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:39.567798    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:41.465934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:41.465898    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:43.206864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:43.206231    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:44.954540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:44.954502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:46.593614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:46.593577    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:46.757921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:46.757884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:47.888890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:47.888856    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:12:48.138760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.138720    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"]
Oct 30 12:12:48.145147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.145081    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.167855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.167821    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-846099-openshift-default-dockercfg-ts844\""
Oct 30 12:12:48.175432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.175397    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"]
Oct 30 12:12:48.234692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.234647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:48.315414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.315370    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.321648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.321610    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.322023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.322004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.322192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.322177    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.322357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.322342    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.322566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.322548    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.322783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.322768    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.322956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.322940    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dlk\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.323122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.323109    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424348    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424410    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dlk\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424491    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424675    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424716    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424798    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424827    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.424903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.425198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.424910    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.432491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.425780    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.432491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.426173    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.432491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.426274    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.432491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.426833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.432491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.427433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.439274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.439243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.442986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.442951    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.451628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.451584    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.451814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.451787    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dlk\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk\") pod \"e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") " pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.477484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.476307    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:48.481883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.481840    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:48.670478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.663525    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:48.831407 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:12:48.831376    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cf13b3_fb7d_474b_b5f2_4a8c4d34bddf.slice/crio-baf32d04f98ce5111c7a68fb5c68df0ec5c4b06122fb1ba0c52d6ead8ec0c82d WatchSource:0}: Error finding container baf32d04f98ce5111c7a68fb5c68df0ec5c4b06122fb1ba0c52d6ead8ec0c82d: Status 404 returned error can't find the container with id baf32d04f98ce5111c7a68fb5c68df0ec5c4b06122fb1ba0c52d6ead8ec0c82d
Oct 30 12:12:48.832574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.832549    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"]
Oct 30 12:12:48.834726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.834692    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:12:48.834852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.834761    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:12:48.834852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:48.834791    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:12:49.009067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:49.009023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" event={"ID":"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf","Type":"ContainerStarted","Data":"9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"}
Oct 30 12:12:49.009712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:49.009686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" event={"ID":"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf","Type":"ContainerStarted","Data":"baf32d04f98ce5111c7a68fb5c68df0ec5c4b06122fb1ba0c52d6ead8ec0c82d"}
Oct 30 12:12:49.057156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:49.057101    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podStartSLOduration=1.057084107 podStartE2EDuration="1.057084107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:49.055811445 +0000 UTC m=+2572.850695940" watchObservedRunningTime="2026-10-30 12:12:49.057084107 +0000 UTC m=+2572.851968594"
Oct 30 12:12:49.477330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:49.477286    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:49.484124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:49.484088    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:49.859529 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:12:49.858773    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:12:50.020766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:50.020725    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:50.021851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:50.021827    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:12:50.471960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:50.471926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:51.501334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:51.501301    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:51.645086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:51.645050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:52.058376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:52.058338    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:52.286890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:52.286854    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:52.431099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:52.430975    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:53.223029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:53.222986    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:53.378820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:53.378782    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:54.398308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:54.398261    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:56.159983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:56.159948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:58.256048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:58.256014    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:12:59.847227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:12:59.847194    5060 scope.go:122] "RemoveContainer" containerID="6787fc22528b21632c428f994100a8ef88f16d4272d85921d956e1760d7161ce"
Oct 30 12:13:00.043229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:00.043198    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:01.743195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:01.741056    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:03.596281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:03.596168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:05.218821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:05.218785    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:05.372308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:05.372272    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:07.165755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.165645    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:07.234017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.233984    5060 generic.go:356] "Generic (PLEG): container finished" podID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerID="3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9" exitCode=0
Oct 30 12:13:07.236140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.234059    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" event={"ID":"892354bd-b329-4d9f-8b3e-a2eda63c55af","Type":"ContainerDied","Data":"3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"}
Oct 30 12:13:07.236140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.234624    5060 scope.go:122] "RemoveContainer" containerID="3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"
Oct 30 12:13:07.236594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.236335    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:07.236594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.236417    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:07.236594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:07.236483    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:08.208478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.203136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:08.244041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.244007    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" event={"ID":"892354bd-b329-4d9f-8b3e-a2eda63c55af","Type":"ContainerStarted","Data":"706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801"}
Oct 30 12:13:08.245701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.245675    5060 generic.go:356] "Generic (PLEG): container finished" podID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerID="0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e" exitCode=0
Oct 30 12:13:08.245860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.245711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" event={"ID":"5689375b-457e-44eb-b779-75c5dd8d35fc","Type":"ContainerDied","Data":"0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"}
Oct 30 12:13:08.246120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.246102    5060 scope.go:122] "RemoveContainer" containerID="0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"
Oct 30 12:13:08.247712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.247684    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:08.247794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.247756    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:08.247794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.247788    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:08.320354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.319946    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 12:13:08.320354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.320007    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 12:13:08.320354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:08.320269    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:13:09.136258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:09.136221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:09.265135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:09.265095    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" event={"ID":"5689375b-457e-44eb-b779-75c5dd8d35fc","Type":"ContainerStarted","Data":"0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231"}
Oct 30 12:13:09.326476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:09.325852    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 12:13:09.326476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:09.325917    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 12:13:09.561129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:09.561097    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:10.241626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:10.241530    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:13:10.242034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:10.242011    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 12:13:10.242225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:10.242201    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 12:13:10.320911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:10.320870    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 12:13:10.321590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:10.321548    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 12:13:10.987171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:10.986177    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:11.241675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:11.241586    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 12:13:11.241936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:11.241910    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 12:13:11.322606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:11.322569    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 12:13:11.323216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:11.323180    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 12:13:12.237807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.237773    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 12:13:12.238057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.238036    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 12:13:12.265720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.265688    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-ca6130"]
Oct 30 12:13:12.265920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.265894    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7380-ca6130" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" containerName="test-curl-7380-ca6130" containerID="cri-o://75c20a08c7575b39a558caa82e91422257cc677aa8bd182e6a5683075d656a1b" gracePeriod=2
Oct 30 12:13:12.270533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.269825    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-ca6130"]
Oct 30 12:13:12.349148 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:13:12.349103    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7380-ca6130\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:13:12.351314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.351273    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 12:13:12.351479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.351331    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 12:13:12.549530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.549426    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"]
Oct 30 12:13:12.550784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.550728    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" containerName="test-curl-7380-ca6130"
Oct 30 12:13:12.550954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.550938    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" containerName="test-curl-7380-ca6130"
Oct 30 12:13:12.552232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.551263    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" containerName="test-curl-7380-ca6130"
Oct 30 12:13:12.570856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.570821    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.573345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.573308    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"]
Oct 30 12:13:12.590331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.590285    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:13:12.590791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.590773    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:13:12.591052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.590997    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:13:12.601700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.601669    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:13:12.692306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.692268    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.696811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.696781    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jmh\" (UniqueName: \"kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.701766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.701733    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.803184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.803093    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.803554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.803530    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.803741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.803725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jmh\" (UniqueName: \"kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.819798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.819766    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.820044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.820020    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.843236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.843202    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jmh\" (UniqueName: \"kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh\") pod \"maas-api-revoke-sub-9036b50a-kt64x\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:12.956702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:12.956668    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:13.039485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.038765    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:13.237479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.237017    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 12:13:13.237479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.237096    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 12:13:13.336479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.321850    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 12:13:13.336479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.321913    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 12:13:13.432527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.432415    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:13:13.437492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:13.437433    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"]
Oct 30 12:13:14.237105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.237056    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 12:13:14.237316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.237146    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 12:13:14.325555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.325524    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:13:14.415511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.415147    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x" event={"ID":"67024988-8e6f-48a3-b4a8-ba3896f18927","Type":"ContainerStarted","Data":"f99410445f791471ee64e53da90d5af1bc4dfe44b2a19b3a164a25da6a24a3d8"}
Oct 30 12:13:14.415511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.415210    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x" event={"ID":"67024988-8e6f-48a3-b4a8-ba3896f18927","Type":"ContainerStarted","Data":"dee0ecc2e6d0aa57fc93bd6c3dd5a213113ce51c511942d3103e5b5d8d04f70d"}
Oct 30 12:13:14.418071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.417973    5060 generic.go:356] "Generic (PLEG): container finished" podID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" containerID="75c20a08c7575b39a558caa82e91422257cc677aa8bd182e6a5683075d656a1b" exitCode=137
Oct 30 12:13:14.420239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.420216    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:13:14.459206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.459133    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x" podStartSLOduration=2.45912204 podStartE2EDuration="2.45912204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:14.458730297 +0000 UTC m=+2598.253614793" watchObservedRunningTime="2026-10-30 12:13:14.45912204 +0000 UTC m=+2598.254006533"
Oct 30 12:13:14.573440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.573414    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:13:14.664216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.664178    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd\" (UniqueName: \"kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd\") pod \"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a\" (UID: \"f28ac3b0-8281-45a1-bc7c-cdeafe72c93a\") "
Oct 30 12:13:14.666540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.666514    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd" pod "f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" (UID: "f28ac3b0-8281-45a1-bc7c-cdeafe72c93a"). InnerVolumeSpecName "kube-api-access-qz4kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:13:14.765819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.765783    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz4kd\" (UniqueName: \"kubernetes.io/projected/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a-kube-api-access-qz4kd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:14.856865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:14.856833    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:15.241164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.241135    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:13:15.241490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.241440    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:13:15.242233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.242214    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:13:15.428537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.428505    5060 generic.go:356] "Generic (PLEG): container finished" podID="67024988-8e6f-48a3-b4a8-ba3896f18927" containerID="f99410445f791471ee64e53da90d5af1bc4dfe44b2a19b3a164a25da6a24a3d8" exitCode=0
Oct 30 12:13:15.428754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.428597    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x" event={"ID":"67024988-8e6f-48a3-b4a8-ba3896f18927","Type":"ContainerDied","Data":"f99410445f791471ee64e53da90d5af1bc4dfe44b2a19b3a164a25da6a24a3d8"}
Oct 30 12:13:15.430057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.430036    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:13:15.430195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:15.430079    5060 scope.go:122] "RemoveContainer" containerID="75c20a08c7575b39a558caa82e91422257cc677aa8bd182e6a5683075d656a1b"
Oct 30 12:13:15.484083 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:13:15.484042    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7380-ca6130\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:13:16.336279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.336228    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" path="/var/lib/kubelet/pods/f28ac3b0-8281-45a1-bc7c-cdeafe72c93a/volumes"
Oct 30 12:13:16.346891 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:13:16.346850    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7380-ca6130\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="f28ac3b0-8281-45a1-bc7c-cdeafe72c93a" pod="opendatahub/test-curl-7380-ca6130"
Oct 30 12:13:16.653197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.653168    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:16.695646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.695588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh\" (UniqueName: \"kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh\") pod \"67024988-8e6f-48a3-b4a8-ba3896f18927\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") "
Oct 30 12:13:16.695825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.695686    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca\") pod \"67024988-8e6f-48a3-b4a8-ba3896f18927\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") "
Oct 30 12:13:16.695825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.695807    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca\") pod \"67024988-8e6f-48a3-b4a8-ba3896f18927\" (UID: \"67024988-8e6f-48a3-b4a8-ba3896f18927\") "
Oct 30 12:13:16.696321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.696292    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca" pod "67024988-8e6f-48a3-b4a8-ba3896f18927" (UID: "67024988-8e6f-48a3-b4a8-ba3896f18927"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:13:16.698332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.698301    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh" pod "67024988-8e6f-48a3-b4a8-ba3896f18927" (UID: "67024988-8e6f-48a3-b4a8-ba3896f18927"). InnerVolumeSpecName "kube-api-access-n8jmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:13:16.699561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.699529    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca" pod "67024988-8e6f-48a3-b4a8-ba3896f18927" (UID: "67024988-8e6f-48a3-b4a8-ba3896f18927"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:13:16.779229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.779181    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:16.797394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.797353    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8jmh\" (UniqueName: \"kubernetes.io/projected/67024988-8e6f-48a3-b4a8-ba3896f18927-kube-api-access-n8jmh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:16.797394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.797389    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:16.797394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:16.797404    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67024988-8e6f-48a3-b4a8-ba3896f18927-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:17.455921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:17.455894    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"
Oct 30 12:13:17.456109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:17.455929    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x" event={"ID":"67024988-8e6f-48a3-b4a8-ba3896f18927","Type":"ContainerDied","Data":"dee0ecc2e6d0aa57fc93bd6c3dd5a213113ce51c511942d3103e5b5d8d04f70d"}
Oct 30 12:13:17.456109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:17.455976    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee0ecc2e6d0aa57fc93bd6c3dd5a213113ce51c511942d3103e5b5d8d04f70d"
Oct 30 12:13:18.637171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:18.637131    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:19.349859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:19.349808    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:19.857619 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:13:19.857586    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:13:20.282736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:20.282696    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:20.564570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:20.564440    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:20.831150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:20.831053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:21.004090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:21.004054    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:21.199095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:21.199062    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:21.375481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:21.375432    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:21.522608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:21.522515    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:22.173631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.173589    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:22.320354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.320301    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:22.939313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.939271    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"]
Oct 30 12:13:22.939924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.939910    5060 cpu_manager.go:414] "containerMap: removing container" podUID="67024988-8e6f-48a3-b4a8-ba3896f18927" containerName="revoke-keys"
Oct 30 12:13:22.939985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.939925    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67024988-8e6f-48a3-b4a8-ba3896f18927" containerName="revoke-keys"
Oct 30 12:13:22.940065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.940056    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="67024988-8e6f-48a3-b4a8-ba3896f18927" containerName="revoke-keys"
Oct 30 12:13:22.943563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.943546    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:22.960368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.960340    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"]
Oct 30 12:13:22.968361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.968332    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:13:22.968609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.968349    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:13:22.968713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.968370    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:13:22.971024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:22.970999    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:13:23.015057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.015012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.015268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.015181    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.015354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.015314    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4cz\" (UniqueName: \"kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.116634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.116601    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.116844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.116709    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.116844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.116777    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4cz\" (UniqueName: \"kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.117607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.117579    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.119351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.119323    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.140292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.140256    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4cz\" (UniqueName: \"kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz\") pod \"maas-api-revoke-sub-6950e800-f2zb6\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.258207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.258169    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:23.547857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.547764    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:23.567196 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:13:23.566228    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30091c2_e6e3_4a8f_a070_c1b4721fcc8e.slice/crio-8bf15ba5179a675b56ae71f564944fce989e824491cb4a51e2ca15c5fcbabae0 WatchSource:0}: Error finding container 8bf15ba5179a675b56ae71f564944fce989e824491cb4a51e2ca15c5fcbabae0: Status 404 returned error can't find the container with id 8bf15ba5179a675b56ae71f564944fce989e824491cb4a51e2ca15c5fcbabae0
Oct 30 12:13:23.569305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.569264    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"]
Oct 30 12:13:23.769813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.769775    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"]
Oct 30 12:13:23.775954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.775925    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:23.796807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.790960    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"agc-cred-78a642bec6ab\""
Oct 30 12:13:23.796807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.793068    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"]
Oct 30 12:13:23.799353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.799286    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2\""
Oct 30 12:13:23.799579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.799341    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"routing-overlay\""
Oct 30 12:13:23.799579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.799377    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"openshift-service-ca.crt\""
Oct 30 12:13:23.799579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.799422    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"kube-root-ca.crt\""
Oct 30 12:13:23.799827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.799499    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"payload-processing-external-model-e2e--793d1056-dockercfg-7lnxq\""
Oct 30 12:13:23.946474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.944283    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:23.946474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.944369    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:23.946474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.944415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7fb\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:23.946474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:23.944859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.046600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.046561    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.046812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.046607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.046812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.046642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.046812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.046670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7fb\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.047368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.047341    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.047502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.047428    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.049309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.049289    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.070318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.070251    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7fb\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.105579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.105547    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.270124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.270086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:24.413311 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:13:24.413276    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode585830e_175e_475c_8598_ce9b4ef4f9d5.slice/crio-77fb5338f906a9cadaf550ea7688c307dba6ea3e0179deeac3e46662fa8bb55f WatchSource:0}: Error finding container 77fb5338f906a9cadaf550ea7688c307dba6ea3e0179deeac3e46662fa8bb55f: Status 404 returned error can't find the container with id 77fb5338f906a9cadaf550ea7688c307dba6ea3e0179deeac3e46662fa8bb55f
Oct 30 12:13:24.414619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.414595    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"]
Oct 30 12:13:24.560815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.560776    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457" event={"ID":"e585830e-175e-475c-8598-ce9b4ef4f9d5","Type":"ContainerStarted","Data":"2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7"}
Oct 30 12:13:24.560992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.560848    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:24.560992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.560874    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457" event={"ID":"e585830e-175e-475c-8598-ce9b4ef4f9d5","Type":"ContainerStarted","Data":"77fb5338f906a9cadaf550ea7688c307dba6ea3e0179deeac3e46662fa8bb55f"}
Oct 30 12:13:24.562640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.562613    5060 generic.go:356] "Generic (PLEG): container finished" podID="d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" containerID="0540915c3d763831fdf4a27a1b066ece75133d390eed29ab4c64017e344991af" exitCode=0
Oct 30 12:13:24.562749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.562705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6" event={"ID":"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e","Type":"ContainerDied","Data":"0540915c3d763831fdf4a27a1b066ece75133d390eed29ab4c64017e344991af"}
Oct 30 12:13:24.562788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.562752    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6" event={"ID":"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e","Type":"ContainerStarted","Data":"8bf15ba5179a675b56ae71f564944fce989e824491cb4a51e2ca15c5fcbabae0"}
Oct 30 12:13:24.602333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:24.602281    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457" podStartSLOduration=1.602264599 podStartE2EDuration="1.602264599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:24.601653808 +0000 UTC m=+2608.396538305" watchObservedRunningTime="2026-10-30 12:13:24.602264599 +0000 UTC m=+2608.397149099"
Oct 30 12:13:25.751584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.751557    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:25.869986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.869903    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz\" (UniqueName: \"kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz\") pod \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") "
Oct 30 12:13:25.870155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.870049    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca\") pod \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") "
Oct 30 12:13:25.870155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.870102    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca\") pod \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\" (UID: \"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e\") "
Oct 30 12:13:25.870547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.870520    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca" pod "d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" (UID: "d30091c2-e6e3-4a8f-a070-c1b4721fcc8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:13:25.872274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.872243    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca" pod "d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" (UID: "d30091c2-e6e3-4a8f-a070-c1b4721fcc8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:13:25.872377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.872330    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz" pod "d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" (UID: "d30091c2-e6e3-4a8f-a070-c1b4721fcc8e"). InnerVolumeSpecName "kube-api-access-kj4cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:13:25.971418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.971381    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4cz\" (UniqueName: \"kubernetes.io/projected/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-kube-api-access-kj4cz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:25.971418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.971419    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:25.971643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:25.971432    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:13:26.022687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:26.022655    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:26.167075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:26.166977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:26.581346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:26.581320    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"
Oct 30 12:13:26.581346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:26.581333    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6" event={"ID":"d30091c2-e6e3-4a8f-a070-c1b4721fcc8e","Type":"ContainerDied","Data":"8bf15ba5179a675b56ae71f564944fce989e824491cb4a51e2ca15c5fcbabae0"}
Oct 30 12:13:26.581625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:26.581377    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf15ba5179a675b56ae71f564944fce989e824491cb4a51e2ca15c5fcbabae0"
Oct 30 12:13:27.801125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:27.801074    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:27.959547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:27.959502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:28.091506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:28.091395    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:29.749622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.749581    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:29.961933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.961899    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"]
Oct 30 12:13:29.962569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.962555    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" containerName="revoke-keys"
Oct 30 12:13:29.962627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.962572    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" containerName="revoke-keys"
Oct 30 12:13:29.962803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.962785    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" containerName="revoke-keys"
Oct 30 12:13:29.966795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.966771    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:29.996923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.996056    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"]
Oct 30 12:13:29.998582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:29.998353    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7b8a9d-openshift-default-dockercfg-rfjd2\""
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.029850    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.029900    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwql\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.029978    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.030054    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.030122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.030210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.030242    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.030297    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.032547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.030323    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.058850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.058818    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:30.131907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.131867    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.131974    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132158    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132192    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132254    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132281    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132343    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132375    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwql\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132546    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132672    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132772    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.132968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.132919    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.134545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.134517    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.134851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.134833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.198267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.198223    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwql\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.200266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.200236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.288806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.288713    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:30.325357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.325306    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:30.668876 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:13:30.668843    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de39bfb_a670_43e2_9880_5466d9662ea7.slice/crio-ae2423ef8ddc43f8cfccec97f56b61e6923dbbbd193126a372e5b603c9ee22cf WatchSource:0}: Error finding container ae2423ef8ddc43f8cfccec97f56b61e6923dbbbd193126a372e5b603c9ee22cf: Status 404 returned error can't find the container with id ae2423ef8ddc43f8cfccec97f56b61e6923dbbbd193126a372e5b603c9ee22cf
Oct 30 12:13:30.670491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.670389    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"]
Oct 30 12:13:30.671639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.671606    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:30.671723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.671671    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:30.671723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:30.671699    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:13:31.560305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:31.560273    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:31.631088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:31.631051    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" event={"ID":"4de39bfb-a670-43e2-9880-5466d9662ea7","Type":"ContainerStarted","Data":"73a54328d8fe3096025b5b03f924a9c47f404217e91508f41245788e2f6ba3a5"}
Oct 30 12:13:31.631276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:31.631114    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" event={"ID":"4de39bfb-a670-43e2-9880-5466d9662ea7","Type":"ContainerStarted","Data":"ae2423ef8ddc43f8cfccec97f56b61e6923dbbbd193126a372e5b603c9ee22cf"}
Oct 30 12:13:31.702317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:31.702252    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podStartSLOduration=2.70223358 podStartE2EDuration="2.70223358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:31.688215409 +0000 UTC m=+2615.483099902" watchObservedRunningTime="2026-10-30 12:13:31.70223358 +0000 UTC m=+2615.497118075"
Oct 30 12:13:32.290509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:32.290465    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:32.296723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:32.296695    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:32.639422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:32.639342    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:32.640334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:32.640316    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:13:33.452734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:33.452700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:34.096993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:34.096953    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:34.248532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:34.248493    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:35.332032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:35.331993    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:35.576520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:35.576485    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:13:35.808891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:35.808849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:35.978727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:35.978690    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:37.248430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:37.248395    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:38.987810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:38.987761    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:40.862444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:40.862401    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:42.645325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:42.645289    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:44.342948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:44.342909    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:45.242559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:45.242526    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"]
Oct 30 12:13:45.295884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:45.295835    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:46.353968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:46.353933    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:48.180806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:48.180768    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:49.862169 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:13:49.862124    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:13:49.886596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:49.886564    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:50.032205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:50.032168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:51.665572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:51.665524    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:52.180331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:52.180291    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:53.538224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:53.538184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:55.237346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:55.237299    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:55.468111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:55.468076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:56.996408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:56.996373    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:13:58.657911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:13:58.657872    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:00.265562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.265514    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" podUID="4094169e-fd19-437c-b51c-f41fb7ad0432" containerName="main" containerID="cri-o://49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c" gracePeriod=45
Oct 30 12:14:00.352240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.352195    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" podUID="4094169e-fd19-437c-b51c-f41fb7ad0432" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": dial tcp 10.132.0.244:8000: connect: connection refused"
Oct 30 12:14:00.446043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.446009    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:00.578027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.577998    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:14:00.676334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676295    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home\") pod \"4094169e-fd19-437c-b51c-f41fb7ad0432\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") "
Oct 30 12:14:00.676566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676430    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs\" (UniqueName: \"kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs\") pod \"4094169e-fd19-437c-b51c-f41fb7ad0432\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") "
Oct 30 12:14:00.676566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676529    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir\") pod \"4094169e-fd19-437c-b51c-f41fb7ad0432\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") "
Oct 30 12:14:00.676680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676566    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache\") pod \"4094169e-fd19-437c-b51c-f41fb7ad0432\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") "
Oct 30 12:14:00.676748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home" pod "4094169e-fd19-437c-b51c-f41fb7ad0432" (UID: "4094169e-fd19-437c-b51c-f41fb7ad0432"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:00.676748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676692    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt\" (UniqueName: \"kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt\") pod \"4094169e-fd19-437c-b51c-f41fb7ad0432\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") "
Oct 30 12:14:00.676926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676897    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir" pod "4094169e-fd19-437c-b51c-f41fb7ad0432" (UID: "4094169e-fd19-437c-b51c-f41fb7ad0432"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:00.677132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676915    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm\") pod \"4094169e-fd19-437c-b51c-f41fb7ad0432\" (UID: \"4094169e-fd19-437c-b51c-f41fb7ad0432\") "
Oct 30 12:14:00.677296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.676921    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache" pod "4094169e-fd19-437c-b51c-f41fb7ad0432" (UID: "4094169e-fd19-437c-b51c-f41fb7ad0432"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:00.677789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.677764    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:00.677899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.677793    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:00.677899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.677809    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:00.679249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.679224    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm" pod "4094169e-fd19-437c-b51c-f41fb7ad0432" (UID: "4094169e-fd19-437c-b51c-f41fb7ad0432"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:00.679511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.679428    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs" pod "4094169e-fd19-437c-b51c-f41fb7ad0432" (UID: "4094169e-fd19-437c-b51c-f41fb7ad0432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:14:00.679599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.679522    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt" pod "4094169e-fd19-437c-b51c-f41fb7ad0432" (UID: "4094169e-fd19-437c-b51c-f41fb7ad0432"). InnerVolumeSpecName "kube-api-access-2mjdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:00.778964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.778932    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4094169e-fd19-437c-b51c-f41fb7ad0432-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:00.778964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.778959    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4094169e-fd19-437c-b51c-f41fb7ad0432-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:00.778964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.778968    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjdt\" (UniqueName: \"kubernetes.io/projected/4094169e-fd19-437c-b51c-f41fb7ad0432-kube-api-access-2mjdt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:00.877712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.877615    5060 generic.go:356] "Generic (PLEG): container finished" podID="4094169e-fd19-437c-b51c-f41fb7ad0432" containerID="49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c" exitCode=0
Oct 30 12:14:00.877712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.877672    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"
Oct 30 12:14:00.877712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.877692    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" event={"ID":"4094169e-fd19-437c-b51c-f41fb7ad0432","Type":"ContainerDied","Data":"49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c"}
Oct 30 12:14:00.878076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.877742    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk" event={"ID":"4094169e-fd19-437c-b51c-f41fb7ad0432","Type":"ContainerDied","Data":"03ee0ca5fb84a913b9eee03aa1ad21fe3a2e0bf9de8179b0a05ec4f596a0b4cb"}
Oct 30 12:14:00.878076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.877764    5060 scope.go:122] "RemoveContainer" containerID="49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c"
Oct 30 12:14:00.890441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.890418    5060 scope.go:122] "RemoveContainer" containerID="49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c"
Oct 30 12:14:00.890953 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:14:00.890926    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c\": container with ID starting with 49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c not found: ID does not exist" containerID="49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c"
Oct 30 12:14:00.891033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.890965    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c"} err="failed to get container status \"49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c\": rpc error: code = NotFound desc = could not find container \"49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c\": container with ID starting with 49bbcced0efcba701e6110b3d605326b45c4498862dc461fe0a4caff9776c45c not found: ID does not exist"
Oct 30 12:14:00.932768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.932735    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"]
Oct 30 12:14:00.944466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:00.944413    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-933195/auth-test-model-933195-kserve-86b7bdc6f-7fmrk"]
Oct 30 12:14:02.118481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:02.118418    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:02.335695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:02.335661    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4094169e-fd19-437c-b51c-f41fb7ad0432" path="/var/lib/kubelet/pods/4094169e-fd19-437c-b51c-f41fb7ad0432/volumes"
Oct 30 12:14:03.863997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:03.863917    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:05.955863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:05.955811    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:06.989598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:06.989562    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:07.428100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.428009    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"]
Oct 30 12:14:07.430798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.430774    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4094169e-fd19-437c-b51c-f41fb7ad0432" containerName="main"
Oct 30 12:14:07.430985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.430972    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4094169e-fd19-437c-b51c-f41fb7ad0432" containerName="main"
Oct 30 12:14:07.431296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.431281    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4094169e-fd19-437c-b51c-f41fb7ad0432" containerName="main"
Oct 30 12:14:07.449338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.449309    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.504812    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-59b8a9-openshift-default-dockercfg-dc44q\""
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.506855    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507074    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507132    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507200    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507234    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfch\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507266    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507306    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.537487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.507348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.608679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.608266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.615271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.615237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.629678    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.629783    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.629848    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.629922    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.629959    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfch\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.629989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.630036    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.630083    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.634848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.635814    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.636069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.641478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.636915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.699527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.647560    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"]
Oct 30 12:14:07.699527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.648183    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.766800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.766749    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfch\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.767437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.767046    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.767915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.767865    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.772830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.772788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token\") pod \"e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") " pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:07.775999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.775981    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:07.931527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.921945    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:07.948441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:07.948408    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:08.239175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.239066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:08.323635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.323608    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"]
Oct 30 12:14:08.330157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.330117    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:08.330289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.330209    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:08.330289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.330254    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:08.962284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.962234    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" event={"ID":"85e29aab-e875-464e-b2e7-3cfb50e236a8","Type":"ContainerStarted","Data":"0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"}
Oct 30 12:14:08.962541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:08.962314    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" event={"ID":"85e29aab-e875-464e-b2e7-3cfb50e236a8","Type":"ContainerStarted","Data":"bcf2d7adddf29311ef5273b73cb6a4bcd87254c2a8dac56b160a78605c0c1df1"}
Oct 30 12:14:09.029761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.029697    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podStartSLOduration=2.029678567 podStartE2EDuration="2.029678567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:09.024446583 +0000 UTC m=+2652.819331080" watchObservedRunningTime="2026-10-30 12:14:09.029678567 +0000 UTC m=+2652.824563062"
Oct 30 12:14:09.073315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.073285    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"]
Oct 30 12:14:09.140601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.135980    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.147224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.146793    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"]
Oct 30 12:14:09.158584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.158559    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3e23ec\"/\"openshift-service-ca.crt\""
Oct 30 12:14:09.159506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.159430    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3e23ec\"/\"kube-root-ca.crt\""
Oct 30 12:14:09.174123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.174090    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:09.211477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.210827    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3e23ec\"/\"autores-model-3e23ec-kserve-self-signed-certs\""
Oct 30 12:14:09.211477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.210956    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3e23ec\"/\"default-dockercfg-kgdcf\""
Oct 30 12:14:09.266298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.266209    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.266740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.266316    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.266740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.266337    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.266740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.266378    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.266740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.266439    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.266740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.266495    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmssn\" (UniqueName: \"kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.372933    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.373660    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.373696    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.373781    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.373889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.373964    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmssn\" (UniqueName: \"kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.374984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.374954    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.381493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.375173    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.401443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.388765    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.409473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.405999    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.418557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.412564    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmssn\" (UniqueName: \"kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.418557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.416733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm\") pod \"autores-model-3e23ec-kserve-75899dfc85-xvgz8\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") " pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.469163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.469128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:09.469820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.469801    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:09.652705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.652659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:09.854654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.851520    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:09.948881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.948768    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:09.959384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:09.959353    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:10.022974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.017957    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:10.115773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.115638    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:10.167159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.166048    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:14:10.171917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.171890    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"]
Oct 30 12:14:10.174605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.174580    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:10.336206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.336179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:10.897479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:10.893652    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:11.060481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:11.049632    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:11.157537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:11.156558    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" event={"ID":"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc","Type":"ContainerStarted","Data":"bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9"}
Oct 30 12:14:11.157537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:11.156615    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" event={"ID":"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc","Type":"ContainerStarted","Data":"1ee6d6a5f8c79ca3a07c7b99b4f3be6db277cef0474fb146aa1ac46db3802525"}
Oct 30 12:14:11.470988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:11.470962    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:11.994199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:11.994162    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:12.162006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:12.161970    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:14.042093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:14.042056    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:14.596729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:14.596691    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:14.750571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:14.750536    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:15.852549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:15.852432    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:16.241979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.241943    5060 generic.go:356] "Generic (PLEG): container finished" podID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerID="9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8" exitCode=0
Oct 30 12:14:16.242192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.242014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" event={"ID":"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf","Type":"ContainerDied","Data":"9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"}
Oct 30 12:14:16.242437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.242425    5060 scope.go:122] "RemoveContainer" containerID="9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"
Oct 30 12:14:16.244232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.244202    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:16.244337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.244272    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:16.244337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.244307    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:16.316696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:16.316634    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" podStartSLOduration=7.316614189 podStartE2EDuration="7.316614189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:11.219735631 +0000 UTC m=+2655.014620126" watchObservedRunningTime="2026-10-30 12:14:16.316614189 +0000 UTC m=+2660.111498684"
Oct 30 12:14:17.251644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:17.251613    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" event={"ID":"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf","Type":"ContainerStarted","Data":"5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd"}
Oct 30 12:14:17.478718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:17.478630    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:14:17.480762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:17.480733    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 12:14:17.481021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:17.480996    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 12:14:17.597078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:17.596983    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:18.488356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:18.487887    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 12:14:18.488356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:18.487953    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 12:14:19.410478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:19.402149    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:19.471130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:19.471094    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:19.498478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:19.480851    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 12:14:19.498478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:19.480948    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 12:14:19.520977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:19.520950    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:19.857857 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:14:19.857823    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:14:20.020607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:20.020559    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:14:20.312650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:20.312621    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:20.483658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:20.483623    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 12:14:20.483941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:20.483914    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 12:14:21.163065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:21.163028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:21.502521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:21.498873    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 12:14:21.502521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:21.498952    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 12:14:22.477467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:22.477420    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 12:14:22.477890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:22.477547    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 12:14:22.947222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:22.947187    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:23.482080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:23.482053    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:14:23.483151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:23.483133    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:14:24.629002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:24.628968    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:26.318750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:26.318712    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:27.934252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:27.934212    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"]
Oct 30 12:14:28.018213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.018179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:28.198058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.197963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:28.250522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.250487    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"]
Oct 30 12:14:28.254980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.254963    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.269796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.269770    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:14:28.270070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.270046    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:14:28.270613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.270560    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:14:28.270827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.270806    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:14:28.272200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.272177    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"]
Oct 30 12:14:28.407133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.407100    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.407342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.407285    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.407414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.407399    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.514806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.514694    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.515042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.514865    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.515042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.514955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.515667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.515642    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.517503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.517477    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.539131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.539099    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2\") pod \"maas-api-revoke-keys-e2e-autores-3e23ec-tct7k\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.573192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.573149    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:28.889634 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:14:28.889599    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode810b04b_a30c_4608_be08_37819cb6ef35.slice/crio-f053eda07baa7e8aead4e30874d492f93cf600a8497fef3ef11f6b2c953c7a74 WatchSource:0}: Error finding container f053eda07baa7e8aead4e30874d492f93cf600a8497fef3ef11f6b2c953c7a74: Status 404 returned error can't find the container with id f053eda07baa7e8aead4e30874d492f93cf600a8497fef3ef11f6b2c953c7a74
Oct 30 12:14:28.893085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:28.893054    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"]
Oct 30 12:14:29.387408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:29.387366    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k" event={"ID":"e810b04b-a30c-4608-be08-37819cb6ef35","Type":"ContainerStarted","Data":"0c30e0a9654879bb27dec5b77005b531a36edf99a4209e59c04462d074aa8806"}
Oct 30 12:14:29.387917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:29.387434    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k" event={"ID":"e810b04b-a30c-4608-be08-37819cb6ef35","Type":"ContainerStarted","Data":"f053eda07baa7e8aead4e30874d492f93cf600a8497fef3ef11f6b2c953c7a74"}
Oct 30 12:14:29.428546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:29.428495    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k" podStartSLOduration=1.428478411 podStartE2EDuration="1.428478411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:29.427060431 +0000 UTC m=+2673.221944924" watchObservedRunningTime="2026-10-30 12:14:29.428478411 +0000 UTC m=+2673.223362904"
Oct 30 12:14:30.093725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:30.093634    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:30.399501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:30.399373    5060 generic.go:356] "Generic (PLEG): container finished" podID="e810b04b-a30c-4608-be08-37819cb6ef35" containerID="0c30e0a9654879bb27dec5b77005b531a36edf99a4209e59c04462d074aa8806" exitCode=0
Oct 30 12:14:30.400008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:30.399489    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k" event={"ID":"e810b04b-a30c-4608-be08-37819cb6ef35","Type":"ContainerDied","Data":"0c30e0a9654879bb27dec5b77005b531a36edf99a4209e59c04462d074aa8806"}
Oct 30 12:14:31.581251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.581223    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:31.758506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.758439    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca\") pod \"e810b04b-a30c-4608-be08-37819cb6ef35\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") "
Oct 30 12:14:31.758743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.758725    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2\") pod \"e810b04b-a30c-4608-be08-37819cb6ef35\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") "
Oct 30 12:14:31.758840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.758821    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca\") pod \"e810b04b-a30c-4608-be08-37819cb6ef35\" (UID: \"e810b04b-a30c-4608-be08-37819cb6ef35\") "
Oct 30 12:14:31.759182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.759156    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca" pod "e810b04b-a30c-4608-be08-37819cb6ef35" (UID: "e810b04b-a30c-4608-be08-37819cb6ef35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:14:31.759308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.759296    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:31.760718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.760692    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca" pod "e810b04b-a30c-4608-be08-37819cb6ef35" (UID: "e810b04b-a30c-4608-be08-37819cb6ef35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:14:31.760881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.760860    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2" pod "e810b04b-a30c-4608-be08-37819cb6ef35" (UID: "e810b04b-a30c-4608-be08-37819cb6ef35"). InnerVolumeSpecName "kube-api-access-tn4q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:31.860629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.860597    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/e810b04b-a30c-4608-be08-37819cb6ef35-kube-api-access-tn4q2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:31.860629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.860627    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e810b04b-a30c-4608-be08-37819cb6ef35-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:31.872675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:31.872640    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:32.418652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:32.418624    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"
Oct 30 12:14:32.418835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:32.418618    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k" event={"ID":"e810b04b-a30c-4608-be08-37819cb6ef35","Type":"ContainerDied","Data":"f053eda07baa7e8aead4e30874d492f93cf600a8497fef3ef11f6b2c953c7a74"}
Oct 30 12:14:32.418835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:32.418770    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f053eda07baa7e8aead4e30874d492f93cf600a8497fef3ef11f6b2c953c7a74"
Oct 30 12:14:33.650838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:33.650788    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:33.810021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:33.809982    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:35.301343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:35.301302    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"]
Oct 30 12:14:35.301865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:35.301571    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy" containerID="cri-o://a8a53dce4d3739d97bc2c550a24981534ac35d65d2090cd6bd5f957d131dcd5d" gracePeriod=30
Oct 30 12:14:35.679117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:35.679023    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:37.270704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:37.270665    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:37.892028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:37.891994    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"]
Oct 30 12:14:37.892215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:37.892189    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy" containerID="cri-o://3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021" gracePeriod=30
Oct 30 12:14:38.740801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:38.740756    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"]
Oct 30 12:14:38.920318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:38.920285    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:39.068424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:39.068333    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:39.435539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:39.434491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:40.650781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.650530    5060 generic.go:356] "Generic (PLEG): container finished" podID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerID="a8a53dce4d3739d97bc2c550a24981534ac35d65d2090cd6bd5f957d131dcd5d" exitCode=0
Oct 30 12:14:40.650781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.650590    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" event={"ID":"cf1481ed-d4fe-4ee8-b254-fb666f62bd93","Type":"ContainerDied","Data":"a8a53dce4d3739d97bc2c550a24981534ac35d65d2090cd6bd5f957d131dcd5d"}
Oct 30 12:14:40.760775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.760750    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:14:40.839017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.838985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839026    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839059    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839102    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839151    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839210    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.839402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.839235    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data\") pod \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\" (UID: \"cf1481ed-d4fe-4ee8-b254-fb666f62bd93\") "
Oct 30 12:14:40.840027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.840009    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:40.840177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.840163    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:40.840575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.840556    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:40.841445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.841422    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:14:40.841999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.841975    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:40.842622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.842595    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:40.849534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.846956    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "kube-api-access-dbvxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:40.849703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.849602    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:14:40.854536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.851805    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy" pod "cf1481ed-d4fe-4ee8-b254-fb666f62bd93" (UID: "cf1481ed-d4fe-4ee8-b254-fb666f62bd93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941557    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941589    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941599    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvxs\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-kube-api-access-dbvxs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941608    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941618    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941627    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941633    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.941641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941641    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:40.942029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:40.941649    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf1481ed-d4fe-4ee8-b254-fb666f62bd93-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:41.261030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:41.260994    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:41.660093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:41.660011    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"
Oct 30 12:14:41.660093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:41.660051    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965" event={"ID":"cf1481ed-d4fe-4ee8-b254-fb666f62bd93","Type":"ContainerDied","Data":"d1b120013684e87e3ffddc557b24739ca4c4a3edd8cf52d99133c76a3c685ed5"}
Oct 30 12:14:41.660650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:41.660117    5060 scope.go:122] "RemoveContainer" containerID="a8a53dce4d3739d97bc2c550a24981534ac35d65d2090cd6bd5f957d131dcd5d"
Oct 30 12:14:41.741789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:41.741757    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"]
Oct 30 12:14:41.753869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:41.753837    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-pq965"]
Oct 30 12:14:42.341776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:42.341741    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" path="/var/lib/kubelet/pods/cf1481ed-d4fe-4ee8-b254-fb666f62bd93/volumes"
Oct 30 12:14:42.411155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:42.411114    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:42.554864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:42.554813    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:42.959976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:42.959931    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" podUID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" containerName="main" containerID="cri-o://bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9" gracePeriod=45
Oct 30 12:14:43.135054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.134985    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:43.317475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.316668    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:43.443757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.442405    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:43.459687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.459613    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:43.466519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.466484    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:14:43.536618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536565    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache\") pod \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") "
Oct 30 12:14:43.536795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536667    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home\") pod \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") "
Oct 30 12:14:43.536795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536789    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir\") pod \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") "
Oct 30 12:14:43.536939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536850    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn\" (UniqueName: \"kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn\") pod \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") "
Oct 30 12:14:43.536939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536863    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache" pod "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" (UID: "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.536939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536880    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs\") pod \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") "
Oct 30 12:14:43.537093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.536991    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir" pod "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" (UID: "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.537093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.537010    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm\") pod \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\" (UID: \"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc\") "
Oct 30 12:14:43.537193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.537103    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home" pod "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" (UID: "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.537669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.537645    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.537755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.537676    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.537755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.537713    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.540789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.540755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs" pod "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" (UID: "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:14:43.541061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.541036    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn" pod "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" (UID: "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc"). InnerVolumeSpecName "kube-api-access-mmssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:43.542020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.541650    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm" pod "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" (UID: "663b5ef2-6e43-45c3-b2b0-c8f5f77314fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.639932    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640025    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640079    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640232    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640259    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.640362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640349    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.641044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640535    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.641044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.640660    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket\") pod \"eebef590-a0f1-49eb-b964-b4dda45ef640\" (UID: \"eebef590-a0f1-49eb-b964-b4dda45ef640\") "
Oct 30 12:14:43.641360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.641303    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.641669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.641364    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmssn\" (UniqueName: \"kubernetes.io/projected/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-kube-api-access-mmssn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.641669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.641395    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.642013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.641699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.642013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.641713    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.642157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.642125    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.642417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.642332    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.642583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.642554    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:14:43.644230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.644197    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "kube-api-access-kbdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:43.645209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.645180    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:14:43.645955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.645931    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:14:43.647198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.647178    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token" pod "eebef590-a0f1-49eb-b964-b4dda45ef640" (UID: "eebef590-a0f1-49eb-b964-b4dda45ef640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:43.686480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.682013    5060 generic.go:356] "Generic (PLEG): container finished" podID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerID="3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021" exitCode=0
Oct 30 12:14:43.686480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.682118    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" event={"ID":"eebef590-a0f1-49eb-b964-b4dda45ef640","Type":"ContainerDied","Data":"3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021"}
Oct 30 12:14:43.686480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.682215    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g" event={"ID":"eebef590-a0f1-49eb-b964-b4dda45ef640","Type":"ContainerDied","Data":"cba54aaba9c4842e74a8cd7954df6b02136a4b3b2a3680f2e6a7b29781a2adb5"}
Oct 30 12:14:43.686480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.682248    5060 scope.go:122] "RemoveContainer" containerID="3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021"
Oct 30 12:14:43.686480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.682399    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"
Oct 30 12:14:43.709671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.701915    5060 generic.go:356] "Generic (PLEG): container finished" podID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" containerID="bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9" exitCode=0
Oct 30 12:14:43.709671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.702255    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" event={"ID":"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc","Type":"ContainerDied","Data":"bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9"}
Oct 30 12:14:43.709671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.702296    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" event={"ID":"663b5ef2-6e43-45c3-b2b0-c8f5f77314fc","Type":"ContainerDied","Data":"1ee6d6a5f8c79ca3a07c7b99b4f3be6db277cef0474fb146aa1ac46db3802525"}
Oct 30 12:14:43.709671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.702403    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"
Oct 30 12:14:43.719193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.719174    5060 scope.go:122] "RemoveContainer" containerID="3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021"
Oct 30 12:14:43.719572 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:14:43.719533    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021\": container with ID starting with 3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021 not found: ID does not exist" containerID="3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021"
Oct 30 12:14:43.719635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.719582    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021"} err="failed to get container status \"3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021\": rpc error: code = NotFound desc = could not find container \"3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021\": container with ID starting with 3edb84cbc787003499b36fb3e8e657a357a61bdfd3d066a80c6b5a2cc97e2021 not found: ID does not exist"
Oct 30 12:14:43.719635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.719601    5060 scope.go:122] "RemoveContainer" containerID="bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9"
Oct 30 12:14:43.730409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.730385    5060 scope.go:122] "RemoveContainer" containerID="bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9"
Oct 30 12:14:43.730794 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:14:43.730774    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9\": container with ID starting with bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9 not found: ID does not exist" containerID="bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9"
Oct 30 12:14:43.730880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.730804    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9"} err="failed to get container status \"bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9\": rpc error: code = NotFound desc = could not find container \"bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9\": container with ID starting with bee8a7f7a4451ead745e4348a0f3d545aa169ae8875ae220cf975442842d9db9 not found: ID does not exist"
Oct 30 12:14:43.741956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.741910    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6" podUID="ebd3967e-2f19-49d2-b38c-bc642f332ddf" containerName="payload-processing" containerID="cri-o://e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8" gracePeriod=25
Oct 30 12:14:43.742919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742892    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742923    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742938    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebef590-a0f1-49eb-b964-b4dda45ef640-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742948    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742956    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742965    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742973    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdp8\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-kube-api-access-kbdp8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742982    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.743029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.742991    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebef590-a0f1-49eb-b964-b4dda45ef640-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:43.795529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.795320    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"]
Oct 30 12:14:43.807641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.807540    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-89c8g"]
Oct 30 12:14:43.869684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.869650    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"]
Oct 30 12:14:43.875596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:43.875567    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8"]
Oct 30 12:14:44.107341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.107019    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:14:44.248860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.248821    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume\") pod \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") "
Oct 30 12:14:44.249050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.248883    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd\" (UniqueName: \"kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd\") pod \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\" (UID: \"ebd3967e-2f19-49d2-b38c-bc642f332ddf\") "
Oct 30 12:14:44.249260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.249225    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume" pod "ebd3967e-2f19-49d2-b38c-bc642f332ddf" (UID: "ebd3967e-2f19-49d2-b38c-bc642f332ddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:14:44.251347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.251307    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd" pod "ebd3967e-2f19-49d2-b38c-bc642f332ddf" (UID: "ebd3967e-2f19-49d2-b38c-bc642f332ddf"). InnerVolumeSpecName "kube-api-access-dxpjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:44.275700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.275663    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-3e23ec/autores-model-3e23ec-kserve-75899dfc85-xvgz8" podUID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:14:44.336379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.336303    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" path="/var/lib/kubelet/pods/663b5ef2-6e43-45c3-b2b0-c8f5f77314fc/volumes"
Oct 30 12:14:44.336867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.336840    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" path="/var/lib/kubelet/pods/eebef590-a0f1-49eb-b964-b4dda45ef640/volumes"
Oct 30 12:14:44.351054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.350991    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3967e-2f19-49d2-b38c-bc642f332ddf-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:44.351054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.351025    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxpjd\" (UniqueName: \"kubernetes.io/projected/ebd3967e-2f19-49d2-b38c-bc642f332ddf-kube-api-access-dxpjd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:44.713921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.713890    5060 generic.go:356] "Generic (PLEG): container finished" podID="ebd3967e-2f19-49d2-b38c-bc642f332ddf" containerID="e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8" exitCode=0
Oct 30 12:14:44.714136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.713937    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"
Oct 30 12:14:44.714136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.713982    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6" event={"ID":"ebd3967e-2f19-49d2-b38c-bc642f332ddf","Type":"ContainerDied","Data":"e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8"}
Oct 30 12:14:44.714136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.714106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6" event={"ID":"ebd3967e-2f19-49d2-b38c-bc642f332ddf","Type":"ContainerDied","Data":"8c1079f2e038b48781647da859f7b6745a78a593b5dd5533f9d62f5b576f0b68"}
Oct 30 12:14:44.714282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.714145    5060 scope.go:122] "RemoveContainer" containerID="e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8"
Oct 30 12:14:44.725980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.725959    5060 scope.go:122] "RemoveContainer" containerID="e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8"
Oct 30 12:14:44.726342 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:14:44.726321    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8\": container with ID starting with e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8 not found: ID does not exist" containerID="e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8"
Oct 30 12:14:44.726424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.726352    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8"} err="failed to get container status \"e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8\": rpc error: code = NotFound desc = could not find container \"e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8\": container with ID starting with e640e492a0f683821b3ff7a76ada5c2ab7a4e8433b54a05ad6a78ca23c9605e8 not found: ID does not exist"
Oct 30 12:14:44.769560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.769527    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"]
Oct 30 12:14:44.781445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:44.781412    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3e23ec-695bdf4db-2g5c6"]
Oct 30 12:14:45.132616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:45.132513    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:46.336977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:46.336940    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd3967e-2f19-49d2-b38c-bc642f332ddf" path="/var/lib/kubelet/pods/ebd3967e-2f19-49d2-b38c-bc642f332ddf/volumes"
Oct 30 12:14:46.838085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:46.838038    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:48.273548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.272394    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": read tcp 10.132.0.2:48156->10.132.1.18:15021: read: connection reset by peer" start-of-body=
Oct 30 12:14:48.273548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.272498    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": read tcp 10.132.0.2:48156->10.132.1.18:15021: read: connection reset by peer"
Oct 30 12:14:48.555821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.555711    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:48.771573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.771540    5060 generic.go:356] "Generic (PLEG): container finished" podID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerID="73a54328d8fe3096025b5b03f924a9c47f404217e91508f41245788e2f6ba3a5" exitCode=0
Oct 30 12:14:48.771750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.771601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" event={"ID":"4de39bfb-a670-43e2-9880-5466d9662ea7","Type":"ContainerDied","Data":"73a54328d8fe3096025b5b03f924a9c47f404217e91508f41245788e2f6ba3a5"}
Oct 30 12:14:48.772024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.772010    5060 scope.go:122] "RemoveContainer" containerID="73a54328d8fe3096025b5b03f924a9c47f404217e91508f41245788e2f6ba3a5"
Oct 30 12:14:48.773878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.773853    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:48.773957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.773917    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:48.773957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:48.773951    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:14:49.034891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:49.034847    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"]
Oct 30 12:14:49.039078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:49.039053    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39064b5d-mwntv"]
Oct 30 12:14:49.799863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:49.799828    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" event={"ID":"4de39bfb-a670-43e2-9880-5466d9662ea7","Type":"ContainerStarted","Data":"54b5996c10ba3d9443b54f1df97c6bca3938539950356e3d7cc91acacd0eafe8"}
Oct 30 12:14:49.858219 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:14:49.858168    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:14:50.318474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:50.317852    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:14:50.318474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:50.318064    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 12:14:50.318474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:50.318117    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 12:14:50.341351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:50.341305    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7" path="/var/lib/kubelet/pods/ee5cdeb1-67fa-4687-a3b8-4d5f8b9746d7/volumes"
Oct 30 12:14:50.409066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:50.409024    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:50.554326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:50.554288    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:51.289860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:51.289821    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 12:14:51.290519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:51.290483    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 12:14:52.293828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:52.293793    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 12:14:52.294468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:52.294416    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 12:14:52.344594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:52.344551    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:52.519476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:52.516272    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:53.292909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.292870    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 12:14:53.293242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.293211    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 12:14:53.910519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.910471    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"]
Oct 30 12:14:53.911179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911163    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ebd3967e-2f19-49d2-b38c-bc642f332ddf" containerName="payload-processing"
Oct 30 12:14:53.911245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911181    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd3967e-2f19-49d2-b38c-bc642f332ddf" containerName="payload-processing"
Oct 30 12:14:53.911245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911193    5060 cpu_manager.go:414] "containerMap: removing container" podUID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" containerName="main"
Oct 30 12:14:53.911245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911198    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" containerName="main"
Oct 30 12:14:53.911245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911206    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e810b04b-a30c-4608-be08-37819cb6ef35" containerName="revoke-keys"
Oct 30 12:14:53.911245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911212    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e810b04b-a30c-4608-be08-37819cb6ef35" containerName="revoke-keys"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911257    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911262    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911272    5060 cpu_manager.go:414] "containerMap: removing container" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911277    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911381    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1481ed-d4fe-4ee8-b254-fb666f62bd93" containerName="istio-proxy"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911396    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="eebef590-a0f1-49eb-b964-b4dda45ef640" containerName="istio-proxy"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911408    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd3967e-2f19-49d2-b38c-bc642f332ddf" containerName="payload-processing"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911414    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b5ef2-6e43-45c3-b2b0-c8f5f77314fc" containerName="main"
Oct 30 12:14:53.911465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.911419    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e810b04b-a30c-4608-be08-37819cb6ef35" containerName="revoke-keys"
Oct 30 12:14:53.915066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.915048    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:53.927026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.926997    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"]
Oct 30 12:14:53.933689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.933650    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:14:53.933901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.933779    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:14:53.934062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.933947    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:14:53.937219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:53.937192    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:14:54.007481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.007378    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgclg\" (UniqueName: \"kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.007481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.007442    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.007736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.007712    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.109388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.109334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgclg\" (UniqueName: \"kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.109388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.109387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.109686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.109567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.110889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.110859    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.114523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.113433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.134531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.134502    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgclg\" (UniqueName: \"kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg\") pod \"maas-api-revoke-sub-0d07102e-4ffvz\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.233415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.233379    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:54.269709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.269681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:54.297939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.297903    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 12:14:54.298247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.298215    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 12:14:54.611758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.611730    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"]
Oct 30 12:14:54.613891 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:14:54.613861    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd5502c_b8c4_442b_9a0f_4ec1b59c3ec3.slice/crio-77b813d71c851019e81fdce5fd537857ebaeb4c0f73c7bed702c84947b8069d3 WatchSource:0}: Error finding container 77b813d71c851019e81fdce5fd537857ebaeb4c0f73c7bed702c84947b8069d3: Status 404 returned error can't find the container with id 77b813d71c851019e81fdce5fd537857ebaeb4c0f73c7bed702c84947b8069d3
Oct 30 12:14:54.854645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:54.854613    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz" event={"ID":"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3","Type":"ContainerStarted","Data":"77b813d71c851019e81fdce5fd537857ebaeb4c0f73c7bed702c84947b8069d3"}
Oct 30 12:14:55.502596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:55.502558    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:14:55.503668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:55.503642    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:14:55.504892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:55.504870    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:14:55.864989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:55.864887    5060 generic.go:356] "Generic (PLEG): container finished" podID="1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" containerID="d981fa8b05883377e21ae253648b1b5d9e00c56c39c2da8d1dccf6f1cd8fbdbe" exitCode=0
Oct 30 12:14:55.865253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:55.865218    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz" event={"ID":"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3","Type":"ContainerDied","Data":"d981fa8b05883377e21ae253648b1b5d9e00c56c39c2da8d1dccf6f1cd8fbdbe"}
Oct 30 12:14:55.883549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:55.883521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:56.067732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:56.067699    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:57.032563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.032539    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:57.076997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.076955    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:57.172698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.172595    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca\") pod \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") "
Oct 30 12:14:57.172889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.172732    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg\" (UniqueName: \"kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg\") pod \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") "
Oct 30 12:14:57.172889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.172810    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca\") pod \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\" (UID: \"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3\") "
Oct 30 12:14:57.173195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.173165    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca" pod "1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" (UID: "1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:14:57.173435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.173355    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:57.175033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.175009    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca" pod "1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" (UID: "1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:14:57.175134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.175092    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg" pod "1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" (UID: "1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3"). InnerVolumeSpecName "kube-api-access-xgclg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:57.222102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.222062    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:57.275297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.275258    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:57.275297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.275292    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgclg\" (UniqueName: \"kubernetes.io/projected/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3-kube-api-access-xgclg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:14:57.367653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.367620    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:57.448374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.448345    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:14:57.466947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.466912    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:14:57.676117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.676079    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:57.883007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.882919    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"
Oct 30 12:14:57.883007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.882939    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz" event={"ID":"1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3","Type":"ContainerDied","Data":"77b813d71c851019e81fdce5fd537857ebaeb4c0f73c7bed702c84947b8069d3"}
Oct 30 12:14:57.883217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.883026    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b813d71c851019e81fdce5fd537857ebaeb4c0f73c7bed702c84947b8069d3"
Oct 30 12:14:57.962009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:57.961975    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:58.125950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:58.125909    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:58.508260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:58.508207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:58.659125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:58.659092    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:59.034820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:59.034787    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"]
Oct 30 12:14:59.046884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:59.046856    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1c78a6-rprgg"]
Oct 30 12:14:59.869712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:59.869675    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:14:59.933244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:14:59.933215    5060 scope.go:122] "RemoveContainer" containerID="44050f3f15071a9e6e38dab335a1d65ef12d66408076a41533bfb3dbbc1f41ae"
Oct 30 12:15:00.032312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.032249    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:00.097747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.097704    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"]
Oct 30 12:15:00.098588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.098564    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" containerName="revoke-keys"
Oct 30 12:15:00.098588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.098587    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" containerName="revoke-keys"
Oct 30 12:15:00.098804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.098782    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" containerName="revoke-keys"
Oct 30 12:15:00.104389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.104366    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.125666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.121581    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"]
Oct 30 12:15:00.130187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.130113    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"openshift-service-ca.crt\""
Oct 30 12:15:00.130187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.130152    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7b8a9d\""
Oct 30 12:15:00.130468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.130226    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"routing-overlay\""
Oct 30 12:15:00.132956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.132931    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-16c43061b500\""
Oct 30 12:15:00.133107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.133025    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"kube-root-ca.crt\""
Oct 30 12:15:00.133107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.133075    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-e2e--b2e93605-dockercfg-8bqs8\""
Oct 30 12:15:00.140467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.140418    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"]
Oct 30 12:15:00.145774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.145744    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.150612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.150582    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"]
Oct 30 12:15:00.168365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.165921    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"]
Oct 30 12:15:00.168365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.167289    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.191312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.190677    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"]
Oct 30 12:15:00.191312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.191234    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.246053    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.246424    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.246566    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.251716    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.251774    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.251809    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5t86\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.266713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.251876    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.357956    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358060    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsc2\" (UniqueName: \"kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlww6\" (UniqueName: \"kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358198    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4xr\" (UniqueName: \"kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358244    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358278    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358311    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5t86\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358388    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358433    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.358572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.359682    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.360496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.360314    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.385507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.369978    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.407202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.407127    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.426492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.424891    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5t86\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.430711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.428502    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eadb3b-8f57-45dc-b82c-89f154769450" path="/var/lib/kubelet/pods/a6eadb3b-8f57-45dc-b82c-89f154769450/volumes"
Oct 30 12:15:00.430711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.428981    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"]
Oct 30 12:15:00.431137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.431107    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:00.441697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.441656    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"]
Oct 30 12:15:00.441850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.441709    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"]
Oct 30 12:15:00.441850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.441736    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"]
Oct 30 12:15:00.442686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.442663    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.448738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.448712    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"]
Oct 30 12:15:00.449030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.448924    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"]
Oct 30 12:15:00.450402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.450213    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.458603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.458578    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"]
Oct 30 12:15:00.458819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.458806    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"]
Oct 30 12:15:00.459281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.458837    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.459960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.459618    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlww6\" (UniqueName: \"kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.459960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.459667    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4xr\" (UniqueName: \"kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.459960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.459715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4p84\" (UniqueName: \"kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.459960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.459796    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.459960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.459899    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.461812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.460416    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.461812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.460481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.461812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.460528    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsc2\" (UniqueName: \"kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.461812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.461641    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.461812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.461733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.462297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.462275    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.464909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.464893    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"]
Oct 30 12:15:00.465119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.465097    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.470072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.470037    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"]
Oct 30 12:15:00.470227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.470137    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.476133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.476103    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"]
Oct 30 12:15:00.476284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.476152    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"]
Oct 30 12:15:00.476284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.476180    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"]
Oct 30 12:15:00.477168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.477142    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.485007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.484979    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsc2\" (UniqueName: \"kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.486389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.486363    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"]
Oct 30 12:15:00.486544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.486413    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"]
Oct 30 12:15:00.486620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.486564    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.488800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.488748    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4xr\" (UniqueName: \"kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.490728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.490703    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlww6\" (UniqueName: \"kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.493213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.493189    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"]
Oct 30 12:15:00.493329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.493241    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"]
Oct 30 12:15:00.493680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.493659    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.498649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.498622    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"]
Oct 30 12:15:00.498790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.498736    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.524445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.524361    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:00.532704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.532651    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"]
Oct 30 12:15:00.532974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.532784    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.553538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.549322    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"]
Oct 30 12:15:00.553538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.549380    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"]
Oct 30 12:15:00.553538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.549524    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.554739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.554715    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"]
Oct 30 12:15:00.561319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.561280    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"]
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.561945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.561991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqv6w\" (UniqueName: \"kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562083    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562141    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562225    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctsz\" (UniqueName: \"kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562323    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562424    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562491    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45xm\" (UniqueName: \"kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562520    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4p84\" (UniqueName: \"kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjtv\" (UniqueName: \"kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562663    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562803    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562839    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5lh\" (UniqueName: \"kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562879    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8wm\" (UniqueName: \"kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.562911    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6kk\" (UniqueName: \"kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.565747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.563235    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.599852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.577379    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"]
Oct 30 12:15:00.603393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.590938    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"]
Oct 30 12:15:00.603659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.603640    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"]
Oct 30 12:15:00.616804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.616770    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4p84\" (UniqueName: \"kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665096    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665548    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5lh\" (UniqueName: \"kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665686    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8wm\" (UniqueName: \"kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665720    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6kk\" (UniqueName: \"kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665759    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665810    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqv6w\" (UniqueName: \"kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665835    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.665911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665862    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889375-stmvk\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.666677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.665979    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.666677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.666677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666663    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctsz\" (UniqueName: \"kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.666905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666691    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7r7\" (UniqueName: \"kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7\") pod \"maas-api-key-cleanup-29889375-stmvk\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.666905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666763    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.666905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666857    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.666905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.666905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666895    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.667191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666983    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.667191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.666999    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.667191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.667032    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.667191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.667073    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45xm\" (UniqueName: \"kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.667191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.667124    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqbw\" (UniqueName: \"kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.669929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.667490    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjtv\" (UniqueName: \"kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.669929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.667699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.669929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.668037    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.669929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.668087    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.681780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.672816    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkqg\" (UniqueName: \"kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg\") pod \"maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.681780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.672872    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.681780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.673688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.701108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.701078    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8wm\" (UniqueName: \"kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.705192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.705156    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctsz\" (UniqueName: \"kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.706420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.706367    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:00.706819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.706768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5lh\" (UniqueName: \"kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.707079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.706994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6kk\" (UniqueName: \"kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.708206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.708047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45xm\" (UniqueName: \"kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.727815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.726770    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjtv\" (UniqueName: \"kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.729054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.728678    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqv6w\" (UniqueName: \"kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.768064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.767992    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775055    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkqg\" (UniqueName: \"kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg\") pod \"maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775342    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775445    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889375-stmvk\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775552    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7r7\" (UniqueName: \"kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7\") pod \"maas-api-key-cleanup-29889375-stmvk\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775694    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775780    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.775859    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqbw\" (UniqueName: \"kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.776243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.776817    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.777045    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889375-stmvk\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.783488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.782278    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.803949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.803871    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkqg\" (UniqueName: \"kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg\") pod \"maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:00.813172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.813103    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqbw\" (UniqueName: \"kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:00.814099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.814038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7r7\" (UniqueName: \"kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7\") pod \"maas-api-key-cleanup-29889375-stmvk\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:00.815641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.814799    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:00.825319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.825200    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"]
Oct 30 12:15:00.853008 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:00.852917    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c28b2d_ffec_4485_810a_72337e356f05.slice/crio-c8386db46c64ab4b893562d5dec030fd9db4d5fb71630f0e13af6247c22fcf28 WatchSource:0}: Error finding container c8386db46c64ab4b893562d5dec030fd9db4d5fb71630f0e13af6247c22fcf28: Status 404 returned error can't find the container with id c8386db46c64ab4b893562d5dec030fd9db4d5fb71630f0e13af6247c22fcf28
Oct 30 12:15:00.871259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.870563    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:00.889064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.889027    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:00.896499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.896445    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"]
Oct 30 12:15:00.900779 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:00.900732    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b19a738_7614_430a_98bc_30bed2518e8a.slice/crio-f3f855d40540c5bbacf36538c4c0042d3c062e3186e8eec1c0d001615a1786bf WatchSource:0}: Error finding container f3f855d40540c5bbacf36538c4c0042d3c062e3186e8eec1c0d001615a1786bf: Status 404 returned error can't find the container with id f3f855d40540c5bbacf36538c4c0042d3c062e3186e8eec1c0d001615a1786bf
Oct 30 12:15:00.903605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.903394    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:00.923958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.923712    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:00.939179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.936048    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:00.958867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.958401    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerStarted","Data":"f3f855d40540c5bbacf36538c4c0042d3c062e3186e8eec1c0d001615a1786bf"}
Oct 30 12:15:00.958867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.958675    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:00.969285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.969163    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh" event={"ID":"84c28b2d-ffec-4485-810a-72337e356f05","Type":"ContainerStarted","Data":"c8386db46c64ab4b893562d5dec030fd9db4d5fb71630f0e13af6247c22fcf28"}
Oct 30 12:15:00.982944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.982730    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:00.988726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:00.987127    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:01.010932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.010698    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:01.027004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.026922    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:01.065048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.050728    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:01.067169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.067086    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"]
Oct 30 12:15:01.113731 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:01.113693    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27208e5c_eac7_4c18_b9e8_b68030349fb9.slice/crio-d8d0fc3f8dbdb345eb401c199b295b9018721e2315f160d83cf7c774ab6a7f7b WatchSource:0}: Error finding container d8d0fc3f8dbdb345eb401c199b295b9018721e2315f160d83cf7c774ab6a7f7b: Status 404 returned error can't find the container with id d8d0fc3f8dbdb345eb401c199b295b9018721e2315f160d83cf7c774ab6a7f7b
Oct 30 12:15:01.125250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.125218    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"]
Oct 30 12:15:01.191504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.191217    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"]
Oct 30 12:15:01.193304 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:01.193277    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83881d6_593b_4926_b2f7_601ef59b98e2.slice/crio-c6fbf5e30e32ad86421b9207a5a3aa38a43470eb80aa92dc9471e5355a79f4e8 WatchSource:0}: Error finding container c6fbf5e30e32ad86421b9207a5a3aa38a43470eb80aa92dc9471e5355a79f4e8: Status 404 returned error can't find the container with id c6fbf5e30e32ad86421b9207a5a3aa38a43470eb80aa92dc9471e5355a79f4e8
Oct 30 12:15:01.358965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.353399    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"]
Oct 30 12:15:01.371871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.368041    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"]
Oct 30 12:15:01.555967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.555940    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"]
Oct 30 12:15:01.574176 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:01.573572    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02e49c9_34d8_44f2_90a3_f4cf680db99b.slice/crio-db275485ff3ec7e6185eb8c2934609cffa1b53fafc22c772c7be3e5c25759d7e WatchSource:0}: Error finding container db275485ff3ec7e6185eb8c2934609cffa1b53fafc22c772c7be3e5c25759d7e: Status 404 returned error can't find the container with id db275485ff3ec7e6185eb8c2934609cffa1b53fafc22c772c7be3e5c25759d7e
Oct 30 12:15:01.578063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.575912    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"]
Oct 30 12:15:01.578063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.576683    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:01.591404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.591363    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"]
Oct 30 12:15:01.795920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.778835    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:01.836757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.816397    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"]
Oct 30 12:15:01.836757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.831852    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"]
Oct 30 12:15:01.871631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.871290    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"]
Oct 30 12:15:01.913772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.906548    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"]
Oct 30 12:15:01.914660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.914630    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"]
Oct 30 12:15:01.927240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.925610    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"]
Oct 30 12:15:01.942770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:01.940331    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"]
Oct 30 12:15:01.944696 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:01.943620    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3fe01a_cd60_4ed4_9a0c_4981a216fc25.slice/crio-6d5d84a9a87e82afddc08a79cef32c46071c9fdcfa0050dd41196ad46880f803 WatchSource:0}: Error finding container 6d5d84a9a87e82afddc08a79cef32c46071c9fdcfa0050dd41196ad46880f803: Status 404 returned error can't find the container with id 6d5d84a9a87e82afddc08a79cef32c46071c9fdcfa0050dd41196ad46880f803
Oct 30 12:15:02.033956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.029057    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerStarted","Data":"ce5b2a5ab8db7ca1bd44ab39594c7b4fa38eda78df03b2af1dd3a77cca7bdd45"}
Oct 30 12:15:02.033956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.032263    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh" event={"ID":"84c28b2d-ffec-4485-810a-72337e356f05","Type":"ContainerStarted","Data":"40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa"}
Oct 30 12:15:02.033956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.033029    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:02.034967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.034936    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerStarted","Data":"4db166aaf89a16cb810d9b4885daab306d34b204505045f8e951c2cabf3b477d"}
Oct 30 12:15:02.090812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.090750    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerStarted","Data":"c5c7a711c53e69166c30df5737db618c17be650402affab08d5599f50091d87c"}
Oct 30 12:15:02.099659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.099632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerStarted","Data":"6d5d84a9a87e82afddc08a79cef32c46071c9fdcfa0050dd41196ad46880f803"}
Oct 30 12:15:02.099814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.099801    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerStarted","Data":"8abeffcb23f590ff68a90bf905c2c5a9397552ca3cb50081656488a02b09ab5e"}
Oct 30 12:15:02.099929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.099917    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerStarted","Data":"a037dd509807cd71107b5f12ba3a0e018e5e0f74ccecfc5270a45978e6a42ed7"}
Oct 30 12:15:02.100021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.100010    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerStarted","Data":"bf37b569ffdff2f74da03c8c3a0d8082c8097164c07b97b38d40de6f1e2697f4"}
Oct 30 12:15:02.113305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.112371    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:02.115534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.114941    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" event={"ID":"b02e49c9-34d8-44f2-90a3-f4cf680db99b","Type":"ContainerStarted","Data":"db275485ff3ec7e6185eb8c2934609cffa1b53fafc22c772c7be3e5c25759d7e"}
Oct 30 12:15:02.118430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.117931    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" event={"ID":"27208e5c-eac7-4c18-b9e8-b68030349fb9","Type":"ContainerStarted","Data":"d8d0fc3f8dbdb345eb401c199b295b9018721e2315f160d83cf7c774ab6a7f7b"}
Oct 30 12:15:02.126882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.126811    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl" event={"ID":"24509a9d-77c1-4139-8f59-c534fd7d1691","Type":"ContainerStarted","Data":"082cb2c8ab96bd1cf20723f789c8ab0f0ee058315af202d7d6865b17dee6447d"}
Oct 30 12:15:02.129181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.129153    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerStarted","Data":"ea2ba5d8526e8e0d62fabfe450f5581c87fb4add0de8cfb9935ae166cc71775f"}
Oct 30 12:15:02.131923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.131698    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" event={"ID":"9e0145c1-1650-4d30-a144-81bde1ed9393","Type":"ContainerStarted","Data":"906b055960b042d993b9c9ef5c444eed45ea7911786dc55e3baa7144119b0b32"}
Oct 30 12:15:02.141659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.138709    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" event={"ID":"c83881d6-593b-4926-b2f7-601ef59b98e2","Type":"ContainerStarted","Data":"c6fbf5e30e32ad86421b9207a5a3aa38a43470eb80aa92dc9471e5355a79f4e8"}
Oct 30 12:15:02.160793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.160135    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh" podStartSLOduration=2.160117906 podStartE2EDuration="2.160117906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:02.159368401 +0000 UTC m=+2705.954252897" watchObservedRunningTime="2026-10-30 12:15:02.160117906 +0000 UTC m=+2705.955002403"
Oct 30 12:15:02.160793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.160228    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" podStartSLOduration=2.160224194 podStartE2EDuration="2.160224194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:02.10609202 +0000 UTC m=+2705.900976518" watchObservedRunningTime="2026-10-30 12:15:02.160224194 +0000 UTC m=+2705.955108711"
Oct 30 12:15:02.161117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.160970    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerStarted","Data":"c88514cc0a079353295a6887f2e09ef832c11e4963b9e508dcd7f38b0ce234e6"}
Oct 30 12:15:02.163686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.163605    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" event={"ID":"88115e62-3f73-48de-bed3-1c5807fcee69","Type":"ContainerStarted","Data":"5e06d41f7c0ea5f31c113d26ff3b37d7fb803b3923dc52866f46c2514fb22b70"}
Oct 30 12:15:02.206811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.206746    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" podStartSLOduration=2.206725845 podStartE2EDuration="2.206725845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:02.205875325 +0000 UTC m=+2706.000759819" watchObservedRunningTime="2026-10-30 12:15:02.206725845 +0000 UTC m=+2706.001610340"
Oct 30 12:15:02.219877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.219840    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"]
Oct 30 12:15:02.224314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.224284    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3e23ec-tct7k"]
Oct 30 12:15:02.261728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.261696    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:02.344069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.344035    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e810b04b-a30c-4608-be08-37819cb6ef35" path="/var/lib/kubelet/pods/e810b04b-a30c-4608-be08-37819cb6ef35/volumes"
Oct 30 12:15:02.408494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.406047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:02.542905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.542813    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:02.768582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:02.768549    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:03.175656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.175617    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerStarted","Data":"bfd5e88ba23d3930a54c9da353bb5db2d70a2343fd86fd00cecfc5b56ce75843"}
Oct 30 12:15:03.182948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.182887    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerStarted","Data":"4e1f772fc5a6db08b3810f40aa1660d757d55b3c95676552d70b989ee659d5f9"}
Oct 30 12:15:03.187985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.187835    5060 generic.go:356] "Generic (PLEG): container finished" podID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerID="50e24ad9b6492aeb93329354379afb2392b252ce0302a4397e84e153d243b721" exitCode=60
Oct 30 12:15:03.187985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.187882    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerDied","Data":"50e24ad9b6492aeb93329354379afb2392b252ce0302a4397e84e153d243b721"}
Oct 30 12:15:03.188254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.188231    5060 scope.go:122] "RemoveContainer" containerID="50e24ad9b6492aeb93329354379afb2392b252ce0302a4397e84e153d243b721"
Oct 30 12:15:03.193969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.193902    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerStarted","Data":"2526dfc804793c8ccf9866d881eabbe79699cee10e9264c58702642af31ca2b2"}
Oct 30 12:15:03.197766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.197739    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerStarted","Data":"5ee38fbd8f5c0a51c20b1564296046deb4260a97a9313f6639e5feb38cbe6e4b"}
Oct 30 12:15:03.199580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.199554    5060 generic.go:356] "Generic (PLEG): container finished" podID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerID="18bdb15fb22967e0de038ac03f1bbdc06a3bbabdb39c01b49d7426522ae974a9" exitCode=60
Oct 30 12:15:03.199689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.199634    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" event={"ID":"b02e49c9-34d8-44f2-90a3-f4cf680db99b","Type":"ContainerDied","Data":"18bdb15fb22967e0de038ac03f1bbdc06a3bbabdb39c01b49d7426522ae974a9"}
Oct 30 12:15:03.199890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.199870    5060 scope.go:122] "RemoveContainer" containerID="18bdb15fb22967e0de038ac03f1bbdc06a3bbabdb39c01b49d7426522ae974a9"
Oct 30 12:15:03.201547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.201521    5060 generic.go:356] "Generic (PLEG): container finished" podID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerID="24b0f8ee2e1b6a5f0484040e88434d25579acb1a03773686e4bbb19ed9fb558c" exitCode=60
Oct 30 12:15:03.201640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.201577    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" event={"ID":"27208e5c-eac7-4c18-b9e8-b68030349fb9","Type":"ContainerDied","Data":"24b0f8ee2e1b6a5f0484040e88434d25579acb1a03773686e4bbb19ed9fb558c"}
Oct 30 12:15:03.201842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.201827    5060 scope.go:122] "RemoveContainer" containerID="24b0f8ee2e1b6a5f0484040e88434d25579acb1a03773686e4bbb19ed9fb558c"
Oct 30 12:15:03.203732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.203703    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl" podUID="24509a9d-77c1-4139-8f59-c534fd7d1691" containerName="cleanup" containerID="cri-o://11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb" gracePeriod=30
Oct 30 12:15:03.203909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.203878    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl" event={"ID":"24509a9d-77c1-4139-8f59-c534fd7d1691","Type":"ContainerStarted","Data":"11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb"}
Oct 30 12:15:03.207396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.206972    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerStarted","Data":"ce21aaff8bcd8caccff9e6d130d12f5cdb53016f101b06cff39b30f550f45f13"}
Oct 30 12:15:03.207396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.207101    5060 scope.go:122] "RemoveContainer" containerID="ce21aaff8bcd8caccff9e6d130d12f5cdb53016f101b06cff39b30f550f45f13"
Oct 30 12:15:03.225356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.225326    5060 generic.go:356] "Generic (PLEG): container finished" podID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerID="f5c553b264bd25b0760044b9da74118a43bffa7e991707c0f06aa475e5184b20" exitCode=60
Oct 30 12:15:03.225516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.225472    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" podStartSLOduration=3.225439277 podStartE2EDuration="3.225439277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.222787967 +0000 UTC m=+2707.017672464" watchObservedRunningTime="2026-10-30 12:15:03.225439277 +0000 UTC m=+2707.020323777"
Oct 30 12:15:03.225655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.225621    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" event={"ID":"9e0145c1-1650-4d30-a144-81bde1ed9393","Type":"ContainerDied","Data":"f5c553b264bd25b0760044b9da74118a43bffa7e991707c0f06aa475e5184b20"}
Oct 30 12:15:03.225797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.225777    5060 scope.go:122] "RemoveContainer" containerID="f5c553b264bd25b0760044b9da74118a43bffa7e991707c0f06aa475e5184b20"
Oct 30 12:15:03.228361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.228335    5060 generic.go:356] "Generic (PLEG): container finished" podID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerID="babbad70fb0cc8b76e4e63120ae8ac4963c0889b96b063a51c498641dc30add4" exitCode=60
Oct 30 12:15:03.228470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.228430    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" event={"ID":"c83881d6-593b-4926-b2f7-601ef59b98e2","Type":"ContainerDied","Data":"babbad70fb0cc8b76e4e63120ae8ac4963c0889b96b063a51c498641dc30add4"}
Oct 30 12:15:03.228924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.228905    5060 scope.go:122] "RemoveContainer" containerID="babbad70fb0cc8b76e4e63120ae8ac4963c0889b96b063a51c498641dc30add4"
Oct 30 12:15:03.230633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.230609    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerStarted","Data":"4b6e341efd422a0c9c33819756e097194e39ff7e03cc9e1c704ab4be26457cc6"}
Oct 30 12:15:03.230826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.230809    5060 scope.go:122] "RemoveContainer" containerID="4b6e341efd422a0c9c33819756e097194e39ff7e03cc9e1c704ab4be26457cc6"
Oct 30 12:15:03.232497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.232477    5060 generic.go:356] "Generic (PLEG): container finished" podID="88115e62-3f73-48de-bed3-1c5807fcee69" containerID="d5766d637fd6e5e8cd30343b4a1c539831e5e37815843b481c3f83a3dacc82be" exitCode=60
Oct 30 12:15:03.232590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.232507    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" event={"ID":"88115e62-3f73-48de-bed3-1c5807fcee69","Type":"ContainerDied","Data":"d5766d637fd6e5e8cd30343b4a1c539831e5e37815843b481c3f83a3dacc82be"}
Oct 30 12:15:03.232852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.232835    5060 scope.go:122] "RemoveContainer" containerID="d5766d637fd6e5e8cd30343b4a1c539831e5e37815843b481c3f83a3dacc82be"
Oct 30 12:15:03.234544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.234508    5060 generic.go:356] "Generic (PLEG): container finished" podID="0b19a738-7614-430a-98bc-30bed2518e8a" containerID="ce5b2a5ab8db7ca1bd44ab39594c7b4fa38eda78df03b2af1dd3a77cca7bdd45" exitCode=60
Oct 30 12:15:03.234638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.234619    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerDied","Data":"ce5b2a5ab8db7ca1bd44ab39594c7b4fa38eda78df03b2af1dd3a77cca7bdd45"}
Oct 30 12:15:03.234818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.234803    5060 scope.go:122] "RemoveContainer" containerID="ce5b2a5ab8db7ca1bd44ab39594c7b4fa38eda78df03b2af1dd3a77cca7bdd45"
Oct 30 12:15:03.236410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.236389    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerStarted","Data":"cbb2d8c3962b11bec6a5129276fe1819db6aad7ee443605806ffc08fcaa49c5e"}
Oct 30 12:15:03.267807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.267764    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl" podStartSLOduration=3.267749917 podStartE2EDuration="3.267749917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.262093926 +0000 UTC m=+2707.056978422" watchObservedRunningTime="2026-10-30 12:15:03.267749917 +0000 UTC m=+2707.062634411"
Oct 30 12:15:03.305935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.305869    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" podStartSLOduration=3.305849924 podStartE2EDuration="3.305849924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.301494011 +0000 UTC m=+2707.096378506" watchObservedRunningTime="2026-10-30 12:15:03.305849924 +0000 UTC m=+2707.100734417"
Oct 30 12:15:03.439138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.439092    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" podStartSLOduration=3.439078353 podStartE2EDuration="3.439078353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.438321719 +0000 UTC m=+2707.233206204" watchObservedRunningTime="2026-10-30 12:15:03.439078353 +0000 UTC m=+2707.233962868"
Oct 30 12:15:03.440895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.440844    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" podStartSLOduration=3.440831601 podStartE2EDuration="3.440831601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.384335353 +0000 UTC m=+2707.179219848" watchObservedRunningTime="2026-10-30 12:15:03.440831601 +0000 UTC m=+2707.235716092"
Oct 30 12:15:03.604765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.604716    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" podStartSLOduration=3.6047021409999997 podStartE2EDuration="3.604702141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.602295665 +0000 UTC m=+2707.397180158" watchObservedRunningTime="2026-10-30 12:15:03.604702141 +0000 UTC m=+2707.399586634"
Oct 30 12:15:03.700885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.700855    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:03.762115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.762086    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca\") pod \"24509a9d-77c1-4139-8f59-c534fd7d1691\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") "
Oct 30 12:15:03.762264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.762228    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg\" (UniqueName: \"kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg\") pod \"24509a9d-77c1-4139-8f59-c534fd7d1691\" (UID: \"24509a9d-77c1-4139-8f59-c534fd7d1691\") "
Oct 30 12:15:03.762484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.762415    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca" pod "24509a9d-77c1-4139-8f59-c534fd7d1691" (UID: "24509a9d-77c1-4139-8f59-c534fd7d1691"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:03.762942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.762921    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509a9d-77c1-4139-8f59-c534fd7d1691-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:03.764672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.764640    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg" pod "24509a9d-77c1-4139-8f59-c534fd7d1691" (UID: "24509a9d-77c1-4139-8f59-c534fd7d1691"). InnerVolumeSpecName "kube-api-access-vrkqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:03.865006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:03.864972    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkqg\" (UniqueName: \"kubernetes.io/projected/24509a9d-77c1-4139-8f59-c534fd7d1691-kube-api-access-vrkqg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:04.212720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.212687    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:04.250732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.250696    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerStarted","Data":"5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3"}
Oct 30 12:15:04.252667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.252639    5060 generic.go:356] "Generic (PLEG): container finished" podID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerID="cbb2d8c3962b11bec6a5129276fe1819db6aad7ee443605806ffc08fcaa49c5e" exitCode=60
Oct 30 12:15:04.252833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.252717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerDied","Data":"cbb2d8c3962b11bec6a5129276fe1819db6aad7ee443605806ffc08fcaa49c5e"}
Oct 30 12:15:04.252976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.252962    5060 scope.go:122] "RemoveContainer" containerID="cbb2d8c3962b11bec6a5129276fe1819db6aad7ee443605806ffc08fcaa49c5e"
Oct 30 12:15:04.254990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.254948    5060 generic.go:356] "Generic (PLEG): container finished" podID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerID="bfd5e88ba23d3930a54c9da353bb5db2d70a2343fd86fd00cecfc5b56ce75843" exitCode=60
Oct 30 12:15:04.255076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.255059    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerDied","Data":"bfd5e88ba23d3930a54c9da353bb5db2d70a2343fd86fd00cecfc5b56ce75843"}
Oct 30 12:15:04.255342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.255324    5060 scope.go:122] "RemoveContainer" containerID="bfd5e88ba23d3930a54c9da353bb5db2d70a2343fd86fd00cecfc5b56ce75843"
Oct 30 12:15:04.258684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.258663    5060 generic.go:356] "Generic (PLEG): container finished" podID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerID="4e1f772fc5a6db08b3810f40aa1660d757d55b3c95676552d70b989ee659d5f9" exitCode=60
Oct 30 12:15:04.258797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.258726    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerDied","Data":"4e1f772fc5a6db08b3810f40aa1660d757d55b3c95676552d70b989ee659d5f9"}
Oct 30 12:15:04.258957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.258939    5060 scope.go:122] "RemoveContainer" containerID="4e1f772fc5a6db08b3810f40aa1660d757d55b3c95676552d70b989ee659d5f9"
Oct 30 12:15:04.261651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.261629    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerStarted","Data":"2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60"}
Oct 30 12:15:04.263377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.263350    5060 generic.go:356] "Generic (PLEG): container finished" podID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerID="2526dfc804793c8ccf9866d881eabbe79699cee10e9264c58702642af31ca2b2" exitCode=60
Oct 30 12:15:04.263537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.263415    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerDied","Data":"2526dfc804793c8ccf9866d881eabbe79699cee10e9264c58702642af31ca2b2"}
Oct 30 12:15:04.264166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.264143    5060 scope.go:122] "RemoveContainer" containerID="2526dfc804793c8ccf9866d881eabbe79699cee10e9264c58702642af31ca2b2"
Oct 30 12:15:04.265832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.265806    5060 generic.go:356] "Generic (PLEG): container finished" podID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerID="5ee38fbd8f5c0a51c20b1564296046deb4260a97a9313f6639e5feb38cbe6e4b" exitCode=60
Oct 30 12:15:04.265937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.265840    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerDied","Data":"5ee38fbd8f5c0a51c20b1564296046deb4260a97a9313f6639e5feb38cbe6e4b"}
Oct 30 12:15:04.266146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.266112    5060 scope.go:122] "RemoveContainer" containerID="5ee38fbd8f5c0a51c20b1564296046deb4260a97a9313f6639e5feb38cbe6e4b"
Oct 30 12:15:04.267555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.267534    5060 generic.go:356] "Generic (PLEG): container finished" podID="24509a9d-77c1-4139-8f59-c534fd7d1691" containerID="11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb" exitCode=6
Oct 30 12:15:04.267670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.267585    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"
Oct 30 12:15:04.267670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.267613    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl" event={"ID":"24509a9d-77c1-4139-8f59-c534fd7d1691","Type":"ContainerDied","Data":"11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb"}
Oct 30 12:15:04.267670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.267655    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl" event={"ID":"24509a9d-77c1-4139-8f59-c534fd7d1691","Type":"ContainerDied","Data":"082cb2c8ab96bd1cf20723f789c8ab0f0ee058315af202d7d6865b17dee6447d"}
Oct 30 12:15:04.267843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.267671    5060 scope.go:122] "RemoveContainer" containerID="11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb"
Oct 30 12:15:04.277305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.277282    5060 generic.go:356] "Generic (PLEG): container finished" podID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerID="ce21aaff8bcd8caccff9e6d130d12f5cdb53016f101b06cff39b30f550f45f13" exitCode=60
Oct 30 12:15:04.277863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.277840    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerDied","Data":"ce21aaff8bcd8caccff9e6d130d12f5cdb53016f101b06cff39b30f550f45f13"}
Oct 30 12:15:04.283969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.283942    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" event={"ID":"c83881d6-593b-4926-b2f7-601ef59b98e2","Type":"ContainerStarted","Data":"2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737"}
Oct 30 12:15:04.288544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.288508    5060 generic.go:356] "Generic (PLEG): container finished" podID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerID="4b6e341efd422a0c9c33819756e097194e39ff7e03cc9e1c704ab4be26457cc6" exitCode=60
Oct 30 12:15:04.288674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.288587    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerDied","Data":"4b6e341efd422a0c9c33819756e097194e39ff7e03cc9e1c704ab4be26457cc6"}
Oct 30 12:15:04.308303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.308182    5060 scope.go:122] "RemoveContainer" containerID="11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb"
Oct 30 12:15:04.309076 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:04.309013    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb\": container with ID starting with 11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb not found: ID does not exist" containerID="11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb"
Oct 30 12:15:04.309076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.309048    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb"} err="failed to get container status \"11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb\": rpc error: code = NotFound desc = could not find container \"11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb\": container with ID starting with 11df9414898c0d51c80b243591eae9543efdb0f7be3d7845ad82bc34ae2772bb not found: ID does not exist"
Oct 30 12:15:04.355478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.345695    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" podStartSLOduration=4.345677148 podStartE2EDuration="4.345677148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:04.335338427 +0000 UTC m=+2708.130222924" watchObservedRunningTime="2026-10-30 12:15:04.345677148 +0000 UTC m=+2708.140561644"
Oct 30 12:15:04.421852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.421793    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" podStartSLOduration=4.421774735 podStartE2EDuration="4.421774735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:04.416549511 +0000 UTC m=+2708.211434005" watchObservedRunningTime="2026-10-30 12:15:04.421774735 +0000 UTC m=+2708.216659230"
Oct 30 12:15:04.455768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.455741    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"]
Oct 30 12:15:04.464232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:04.464202    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-3e23ec-29889375-lnjcl"]
Oct 30 12:15:05.302827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.302789    5060 generic.go:356] "Generic (PLEG): container finished" podID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerID="8908cfa5631eebc366924cc85b977b171e7e38534d2be8e8df5a62fdaa5b736c" exitCode=60
Oct 30 12:15:05.303339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.302890    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" event={"ID":"9e0145c1-1650-4d30-a144-81bde1ed9393","Type":"ContainerDied","Data":"8908cfa5631eebc366924cc85b977b171e7e38534d2be8e8df5a62fdaa5b736c"}
Oct 30 12:15:05.303339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.302946    5060 scope.go:122] "RemoveContainer" containerID="f5c553b264bd25b0760044b9da74118a43bffa7e991707c0f06aa475e5184b20"
Oct 30 12:15:05.303339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.303309    5060 scope.go:122] "RemoveContainer" containerID="8908cfa5631eebc366924cc85b977b171e7e38534d2be8e8df5a62fdaa5b736c"
Oct 30 12:15:05.303749 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.303724    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql_odh-ai-gateway-infra(9e0145c1-1650-4d30-a144-81bde1ed9393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393"
Oct 30 12:15:05.306044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.306017    5060 generic.go:356] "Generic (PLEG): container finished" podID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerID="2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737" exitCode=60
Oct 30 12:15:05.306167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.306097    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" event={"ID":"c83881d6-593b-4926-b2f7-601ef59b98e2","Type":"ContainerDied","Data":"2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737"}
Oct 30 12:15:05.306515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.306496    5060 scope.go:122] "RemoveContainer" containerID="2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737"
Oct 30 12:15:05.306953 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.306929    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp_odh-ai-gateway-infra(c83881d6-593b-4926-b2f7-601ef59b98e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2"
Oct 30 12:15:05.308964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.308939    5060 generic.go:356] "Generic (PLEG): container finished" podID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerID="9e62a3b9f0666303323d23937c49e27022579fc34d3cb47525f3929ca5bc5acf" exitCode=60
Oct 30 12:15:05.309088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.309029    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerDied","Data":"9e62a3b9f0666303323d23937c49e27022579fc34d3cb47525f3929ca5bc5acf"}
Oct 30 12:15:05.309408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.309385    5060 scope.go:122] "RemoveContainer" containerID="9e62a3b9f0666303323d23937c49e27022579fc34d3cb47525f3929ca5bc5acf"
Oct 30 12:15:05.309827 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.309790    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4_odh-ai-gateway-infra(600d3b71-9d75-4bf8-8e63-d54049d6334e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e"
Oct 30 12:15:05.311701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.311675    5060 generic.go:356] "Generic (PLEG): container finished" podID="88115e62-3f73-48de-bed3-1c5807fcee69" containerID="76f91a5ab17a1dc95d08324477546918761cdfd24867ce955f0880a525f7fdce" exitCode=60
Oct 30 12:15:05.311799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.311759    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" event={"ID":"88115e62-3f73-48de-bed3-1c5807fcee69","Type":"ContainerDied","Data":"76f91a5ab17a1dc95d08324477546918761cdfd24867ce955f0880a525f7fdce"}
Oct 30 12:15:05.312096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.312077    5060 scope.go:122] "RemoveContainer" containerID="76f91a5ab17a1dc95d08324477546918761cdfd24867ce955f0880a525f7fdce"
Oct 30 12:15:05.312508 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.312430    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6_odh-ai-gateway-infra(88115e62-3f73-48de-bed3-1c5807fcee69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" podUID="88115e62-3f73-48de-bed3-1c5807fcee69"
Oct 30 12:15:05.314248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.314225    5060 generic.go:356] "Generic (PLEG): container finished" podID="0b19a738-7614-430a-98bc-30bed2518e8a" containerID="5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3" exitCode=60
Oct 30 12:15:05.314389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.314363    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerDied","Data":"5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3"}
Oct 30 12:15:05.314666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.314647    5060 scope.go:122] "RemoveContainer" containerID="5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3"
Oct 30 12:15:05.315112 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.314994    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9_odh-ai-gateway-infra(0b19a738-7614-430a-98bc-30bed2518e8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" podUID="0b19a738-7614-430a-98bc-30bed2518e8a"
Oct 30 12:15:05.317361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.316762    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerStarted","Data":"19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a"}
Oct 30 12:15:05.321059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.321000    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerStarted","Data":"5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5"}
Oct 30 12:15:05.323607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.323540    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerStarted","Data":"5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb"}
Oct 30 12:15:05.326299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.326275    5060 generic.go:356] "Generic (PLEG): container finished" podID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerID="2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60" exitCode=60
Oct 30 12:15:05.326424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.326369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerDied","Data":"2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60"}
Oct 30 12:15:05.326714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.326695    5060 scope.go:122] "RemoveContainer" containerID="2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60"
Oct 30 12:15:05.326975 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.326953    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z_odh-ai-gateway-infra(b511f39e-f4b2-494e-86ad-9c4a66b8920b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b"
Oct 30 12:15:05.329941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.329913    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerStarted","Data":"309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c"}
Oct 30 12:15:05.332989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.332962    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerStarted","Data":"454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e"}
Oct 30 12:15:05.333602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.333583    5060 scope.go:122] "RemoveContainer" containerID="babbad70fb0cc8b76e4e63120ae8ac4963c0889b96b063a51c498641dc30add4"
Oct 30 12:15:05.335841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.335817    5060 generic.go:356] "Generic (PLEG): container finished" podID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerID="92f156b8842cfae503700b4f91b838cd8ee143c4aced2b731c65cc6ce2ece5b3" exitCode=60
Oct 30 12:15:05.335956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.335912    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" event={"ID":"b02e49c9-34d8-44f2-90a3-f4cf680db99b","Type":"ContainerDied","Data":"92f156b8842cfae503700b4f91b838cd8ee143c4aced2b731c65cc6ce2ece5b3"}
Oct 30 12:15:05.336245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.336228    5060 scope.go:122] "RemoveContainer" containerID="92f156b8842cfae503700b4f91b838cd8ee143c4aced2b731c65cc6ce2ece5b3"
Oct 30 12:15:05.336555 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.336503    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc_odh-ai-gateway-infra(b02e49c9-34d8-44f2-90a3-f4cf680db99b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b"
Oct 30 12:15:05.349203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.348062    5060 generic.go:356] "Generic (PLEG): container finished" podID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerID="b4dc620bc821078208a60606c22f46753ab8cdb260d1b3dfa325e1002380a955" exitCode=60
Oct 30 12:15:05.349203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.348199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" event={"ID":"27208e5c-eac7-4c18-b9e8-b68030349fb9","Type":"ContainerDied","Data":"b4dc620bc821078208a60606c22f46753ab8cdb260d1b3dfa325e1002380a955"}
Oct 30 12:15:05.349203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.348720    5060 scope.go:122] "RemoveContainer" containerID="b4dc620bc821078208a60606c22f46753ab8cdb260d1b3dfa325e1002380a955"
Oct 30 12:15:05.349203 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.349129    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6_odh-ai-gateway-infra(27208e5c-eac7-4c18-b9e8-b68030349fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9"
Oct 30 12:15:05.354390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.353848    5060 generic.go:356] "Generic (PLEG): container finished" podID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerID="e5d7079c5c9bd7e55ffd7f4e719c5ba68f1162238bba14a620dd5dfdee19cbde" exitCode=60
Oct 30 12:15:05.354390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.353895    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerDied","Data":"e5d7079c5c9bd7e55ffd7f4e719c5ba68f1162238bba14a620dd5dfdee19cbde"}
Oct 30 12:15:05.354390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.354348    5060 scope.go:122] "RemoveContainer" containerID="e5d7079c5c9bd7e55ffd7f4e719c5ba68f1162238bba14a620dd5dfdee19cbde"
Oct 30 12:15:05.355142 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:05.355085    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx_odh-ai-gateway-infra(a70afb93-c0d8-430b-8d25-8cf630003fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5"
Oct 30 12:15:05.365695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.365586    5060 scope.go:122] "RemoveContainer" containerID="4b6e341efd422a0c9c33819756e097194e39ff7e03cc9e1c704ab4be26457cc6"
Oct 30 12:15:05.372827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.372800    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"]
Oct 30 12:15:05.401101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.401010    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:05.421579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.421091    5060 scope.go:122] "RemoveContainer" containerID="d5766d637fd6e5e8cd30343b4a1c539831e5e37815843b481c3f83a3dacc82be"
Oct 30 12:15:05.442752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.442731    5060 scope.go:122] "RemoveContainer" containerID="ce5b2a5ab8db7ca1bd44ab39594c7b4fa38eda78df03b2af1dd3a77cca7bdd45"
Oct 30 12:15:05.456573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.456542    5060 scope.go:122] "RemoveContainer" containerID="50e24ad9b6492aeb93329354379afb2392b252ce0302a4397e84e153d243b721"
Oct 30 12:15:05.474752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.474653    5060 scope.go:122] "RemoveContainer" containerID="18bdb15fb22967e0de038ac03f1bbdc06a3bbabdb39c01b49d7426522ae974a9"
Oct 30 12:15:05.495909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.495885    5060 scope.go:122] "RemoveContainer" containerID="24b0f8ee2e1b6a5f0484040e88434d25579acb1a03773686e4bbb19ed9fb558c"
Oct 30 12:15:05.515279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.515250    5060 scope.go:122] "RemoveContainer" containerID="ce21aaff8bcd8caccff9e6d130d12f5cdb53016f101b06cff39b30f550f45f13"
Oct 30 12:15:05.556974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:05.556938    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:06.056819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.056716    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:06.201788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.201751    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:06.339243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.339167    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24509a9d-77c1-4139-8f59-c534fd7d1691" path="/var/lib/kubelet/pods/24509a9d-77c1-4139-8f59-c534fd7d1691/volumes"
Oct 30 12:15:06.365378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.365344    5060 generic.go:356] "Generic (PLEG): container finished" podID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerID="19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a" exitCode=60
Oct 30 12:15:06.365609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.365430    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerDied","Data":"19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a"}
Oct 30 12:15:06.365609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.365537    5060 scope.go:122] "RemoveContainer" containerID="cbb2d8c3962b11bec6a5129276fe1819db6aad7ee443605806ffc08fcaa49c5e"
Oct 30 12:15:06.365797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.365782    5060 scope.go:122] "RemoveContainer" containerID="19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a"
Oct 30 12:15:06.366191 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.366164    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns_odh-ai-gateway-infra(e38aaaa9-f3bf-42db-8e40-47761f7b3c3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a"
Oct 30 12:15:06.367724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.367700    5060 generic.go:356] "Generic (PLEG): container finished" podID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerID="5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5" exitCode=60
Oct 30 12:15:06.367842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.367768    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerDied","Data":"5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5"}
Oct 30 12:15:06.368203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.368182    5060 scope.go:122] "RemoveContainer" containerID="5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5"
Oct 30 12:15:06.368575 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.368549    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws_odh-ai-gateway-infra(7ea959e0-fac6-4c6c-96c1-5e106c45e3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9"
Oct 30 12:15:06.369954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.369934    5060 generic.go:356] "Generic (PLEG): container finished" podID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerID="5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb" exitCode=60
Oct 30 12:15:06.370055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.369962    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerDied","Data":"5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb"}
Oct 30 12:15:06.370251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.370234    5060 scope.go:122] "RemoveContainer" containerID="5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb"
Oct 30 12:15:06.370558 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.370540    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g_odh-ai-gateway-infra(6e3fe01a-cd60-4ed4-9a0c-4981a216fc25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25"
Oct 30 12:15:06.372425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.372407    5060 scope.go:122] "RemoveContainer" containerID="2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60"
Oct 30 12:15:06.372696 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.372673    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z_odh-ai-gateway-infra(b511f39e-f4b2-494e-86ad-9c4a66b8920b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b"
Oct 30 12:15:06.374198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.374175    5060 generic.go:356] "Generic (PLEG): container finished" podID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerID="309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c" exitCode=60
Oct 30 12:15:06.374301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.374272    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerDied","Data":"309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c"}
Oct 30 12:15:06.374880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.374867    5060 scope.go:122] "RemoveContainer" containerID="309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c"
Oct 30 12:15:06.375107 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.375084    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889375-stmvk_odh-ai-gateway-infra(6813457a-74d2-4a5c-8b96-0e0fa0909751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751"
Oct 30 12:15:06.376354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.376336    5060 generic.go:356] "Generic (PLEG): container finished" podID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerID="454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e" exitCode=60
Oct 30 12:15:06.376476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.376399    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerDied","Data":"454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e"}
Oct 30 12:15:06.376688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.376671    5060 scope.go:122] "RemoveContainer" containerID="454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e"
Oct 30 12:15:06.376923 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.376905    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2_odh-ai-gateway-infra(5b8cbfdb-36d8-4f32-80ae-ad130c47817e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e"
Oct 30 12:15:06.378422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.378404    5060 scope.go:122] "RemoveContainer" containerID="92f156b8842cfae503700b4f91b838cd8ee143c4aced2b731c65cc6ce2ece5b3"
Oct 30 12:15:06.378655 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.378638    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc_odh-ai-gateway-infra(b02e49c9-34d8-44f2-90a3-f4cf680db99b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b"
Oct 30 12:15:06.380063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.380046    5060 scope.go:122] "RemoveContainer" containerID="b4dc620bc821078208a60606c22f46753ab8cdb260d1b3dfa325e1002380a955"
Oct 30 12:15:06.380252 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.380229    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6_odh-ai-gateway-infra(27208e5c-eac7-4c18-b9e8-b68030349fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9"
Oct 30 12:15:06.381828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.381809    5060 scope.go:122] "RemoveContainer" containerID="e5d7079c5c9bd7e55ffd7f4e719c5ba68f1162238bba14a620dd5dfdee19cbde"
Oct 30 12:15:06.382030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.382011    5060 scope.go:122] "RemoveContainer" containerID="bfd5e88ba23d3930a54c9da353bb5db2d70a2343fd86fd00cecfc5b56ce75843"
Oct 30 12:15:06.382076 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.382033    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx_odh-ai-gateway-infra(a70afb93-c0d8-430b-8d25-8cf630003fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5"
Oct 30 12:15:06.383641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.383616    5060 scope.go:122] "RemoveContainer" containerID="8908cfa5631eebc366924cc85b977b171e7e38534d2be8e8df5a62fdaa5b736c"
Oct 30 12:15:06.383850 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.383830    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql_odh-ai-gateway-infra(9e0145c1-1650-4d30-a144-81bde1ed9393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393"
Oct 30 12:15:06.385376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.385358    5060 scope.go:122] "RemoveContainer" containerID="2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737"
Oct 30 12:15:06.385606 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.385585    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp_odh-ai-gateway-infra(c83881d6-593b-4926-b2f7-601ef59b98e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2"
Oct 30 12:15:06.386963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.386946    5060 scope.go:122] "RemoveContainer" containerID="9e62a3b9f0666303323d23937c49e27022579fc34d3cb47525f3929ca5bc5acf"
Oct 30 12:15:06.387157 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.387137    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4_odh-ai-gateway-infra(600d3b71-9d75-4bf8-8e63-d54049d6334e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e"
Oct 30 12:15:06.388779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.388757    5060 scope.go:122] "RemoveContainer" containerID="76f91a5ab17a1dc95d08324477546918761cdfd24867ce955f0880a525f7fdce"
Oct 30 12:15:06.389007 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.388987    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6_odh-ai-gateway-infra(88115e62-3f73-48de-bed3-1c5807fcee69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" podUID="88115e62-3f73-48de-bed3-1c5807fcee69"
Oct 30 12:15:06.390601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.390578    5060 scope.go:122] "RemoveContainer" containerID="5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3"
Oct 30 12:15:06.390821 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:06.390802    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9_odh-ai-gateway-infra(0b19a738-7614-430a-98bc-30bed2518e8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" podUID="0b19a738-7614-430a-98bc-30bed2518e8a"
Oct 30 12:15:06.397176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.397157    5060 scope.go:122] "RemoveContainer" containerID="4e1f772fc5a6db08b3810f40aa1660d757d55b3c95676552d70b989ee659d5f9"
Oct 30 12:15:06.411718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.411691    5060 scope.go:122] "RemoveContainer" containerID="2526dfc804793c8ccf9866d881eabbe79699cee10e9264c58702642af31ca2b2"
Oct 30 12:15:06.427248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:06.427224    5060 scope.go:122] "RemoveContainer" containerID="5ee38fbd8f5c0a51c20b1564296046deb4260a97a9313f6639e5feb38cbe6e4b"
Oct 30 12:15:07.401230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:07.401199    5060 scope.go:122] "RemoveContainer" containerID="19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a"
Oct 30 12:15:07.401693 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:07.401606    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns_odh-ai-gateway-infra(e38aaaa9-f3bf-42db-8e40-47761f7b3c3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a"
Oct 30 12:15:07.402985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:07.402963    5060 scope.go:122] "RemoveContainer" containerID="5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5"
Oct 30 12:15:07.403302 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:07.403278    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws_odh-ai-gateway-infra(7ea959e0-fac6-4c6c-96c1-5e106c45e3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9"
Oct 30 12:15:07.404868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:07.404850    5060 scope.go:122] "RemoveContainer" containerID="5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb"
Oct 30 12:15:07.405044 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:07.405026    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g_odh-ai-gateway-infra(6e3fe01a-cd60-4ed4-9a0c-4981a216fc25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25"
Oct 30 12:15:07.406970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:07.406951    5060 scope.go:122] "RemoveContainer" containerID="309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c"
Oct 30 12:15:07.407197 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:07.407162    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889375-stmvk_odh-ai-gateway-infra(6813457a-74d2-4a5c-8b96-0e0fa0909751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751"
Oct 30 12:15:07.408659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:07.408642    5060 scope.go:122] "RemoveContainer" containerID="454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e"
Oct 30 12:15:07.408890 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:07.408870    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2_odh-ai-gateway-infra(5b8cbfdb-36d8-4f32-80ae-ad130c47817e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e"
Oct 30 12:15:08.044206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:08.044166    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:09.218944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:09.218910    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:09.384909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:09.384871    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:09.582711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:09.582614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:09.906130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:09.906047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:10.374308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.374256    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457" podUID="e585830e-175e-475c-8598-ce9b4ef4f9d5" containerName="payload-processing" containerID="cri-o://2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7" gracePeriod=25
Oct 30 12:15:10.690014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.689985    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:15:10.781994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.781955    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials\") pod \"e585830e-175e-475c-8598-ce9b4ef4f9d5\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") "
Oct 30 12:15:10.782236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.782150    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume\") pod \"e585830e-175e-475c-8598-ce9b4ef4f9d5\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") "
Oct 30 12:15:10.782385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.782362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb\") pod \"e585830e-175e-475c-8598-ce9b4ef4f9d5\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") "
Oct 30 12:15:10.782510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.782492    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume\") pod \"e585830e-175e-475c-8598-ce9b4ef4f9d5\" (UID: \"e585830e-175e-475c-8598-ce9b4ef4f9d5\") "
Oct 30 12:15:10.782755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.782725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume" pod "e585830e-175e-475c-8598-ce9b4ef4f9d5" (UID: "e585830e-175e-475c-8598-ce9b4ef4f9d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:10.783205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.783178    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:10.783648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.783620    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume" pod "e585830e-175e-475c-8598-ce9b4ef4f9d5" (UID: "e585830e-175e-475c-8598-ce9b4ef4f9d5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:10.786082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.786045    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials" pod "e585830e-175e-475c-8598-ce9b4ef4f9d5" (UID: "e585830e-175e-475c-8598-ce9b4ef4f9d5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:10.787508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.787432    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb" pod "e585830e-175e-475c-8598-ce9b4ef4f9d5" (UID: "e585830e-175e-475c-8598-ce9b4ef4f9d5"). InnerVolumeSpecName "kube-api-access-2t7fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:10.884463 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.884429    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7fb\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-kube-api-access-2t7fb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:10.884645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.884472    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e585830e-175e-475c-8598-ce9b4ef4f9d5-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:10.884645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:10.884485    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e585830e-175e-475c-8598-ce9b4ef4f9d5-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:11.458858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.458817    5060 generic.go:356] "Generic (PLEG): container finished" podID="e585830e-175e-475c-8598-ce9b4ef4f9d5" containerID="2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7" exitCode=0
Oct 30 12:15:11.459306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.458872    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"
Oct 30 12:15:11.459306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.458894    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457" event={"ID":"e585830e-175e-475c-8598-ce9b4ef4f9d5","Type":"ContainerDied","Data":"2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7"}
Oct 30 12:15:11.459306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.458965    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457" event={"ID":"e585830e-175e-475c-8598-ce9b4ef4f9d5","Type":"ContainerDied","Data":"77fb5338f906a9cadaf550ea7688c307dba6ea3e0179deeac3e46662fa8bb55f"}
Oct 30 12:15:11.459306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.458988    5060 scope.go:122] "RemoveContainer" containerID="2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7"
Oct 30 12:15:11.470369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.470348    5060 scope.go:122] "RemoveContainer" containerID="2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7"
Oct 30 12:15:11.470653 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:11.470634    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7\": container with ID starting with 2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7 not found: ID does not exist" containerID="2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7"
Oct 30 12:15:11.470708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.470664    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7"} err="failed to get container status \"2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7\": rpc error: code = NotFound desc = could not find container \"2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7\": container with ID starting with 2bf970d47c090e819c322b6a626ce7864bac254c4cc9ab98daa4c976930819a7 not found: ID does not exist"
Oct 30 12:15:11.504701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.504661    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:11.512994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.512826    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"]
Oct 30 12:15:11.524870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.524843    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-69f8lj457"]
Oct 30 12:15:11.700574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:11.700537    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:12.240161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:12.240129    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:12.336979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:12.336945    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e585830e-175e-475c-8598-ce9b4ef4f9d5" path="/var/lib/kubelet/pods/e585830e-175e-475c-8598-ce9b4ef4f9d5/volumes"
Oct 30 12:15:12.386625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:12.386593    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:12.530541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:12.530435    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:13.384873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:13.384825    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:13.523516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:13.523431    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:14.293122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:14.293084    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:15:14.294315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:14.294285    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:15.170365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:15.170333    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:15.328034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:15.328005    5060 scope.go:122] "RemoveContainer" containerID="2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737"
Oct 30 12:15:16.332828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.332797    5060 scope.go:122] "RemoveContainer" containerID="8908cfa5631eebc366924cc85b977b171e7e38534d2be8e8df5a62fdaa5b736c"
Oct 30 12:15:16.333241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.332998    5060 scope.go:122] "RemoveContainer" containerID="e5d7079c5c9bd7e55ffd7f4e719c5ba68f1162238bba14a620dd5dfdee19cbde"
Oct 30 12:15:16.333291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.333272    5060 scope.go:122] "RemoveContainer" containerID="5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3"
Oct 30 12:15:16.333372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.333353    5060 scope.go:122] "RemoveContainer" containerID="5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb"
Oct 30 12:15:16.512187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.512147    5060 generic.go:356] "Generic (PLEG): container finished" podID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerID="a84f93d07b0c754941db58b8f4c5aeca07f7c03931536043a20579267e18c62e" exitCode=60
Oct 30 12:15:16.512354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.512325    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" event={"ID":"c83881d6-593b-4926-b2f7-601ef59b98e2","Type":"ContainerDied","Data":"a84f93d07b0c754941db58b8f4c5aeca07f7c03931536043a20579267e18c62e"}
Oct 30 12:15:16.512400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.512367    5060 scope.go:122] "RemoveContainer" containerID="2709660d85cf4a4e26a66ec0f4e4eaee9c7d89755fbb49b8c64d6c597806d737"
Oct 30 12:15:16.512982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.512958    5060 scope.go:122] "RemoveContainer" containerID="a84f93d07b0c754941db58b8f4c5aeca07f7c03931536043a20579267e18c62e"
Oct 30 12:15:16.513371 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:16.513344    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp_odh-ai-gateway-infra(c83881d6-593b-4926-b2f7-601ef59b98e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2"
Oct 30 12:15:16.853330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:16.853286    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:17.328327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.328219    5060 scope.go:122] "RemoveContainer" containerID="19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a"
Oct 30 12:15:17.328571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.328355    5060 scope.go:122] "RemoveContainer" containerID="454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e"
Oct 30 12:15:17.329124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.329088    5060 scope.go:122] "RemoveContainer" containerID="b4dc620bc821078208a60606c22f46753ab8cdb260d1b3dfa325e1002380a955"
Oct 30 12:15:17.525513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.525481    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerStarted","Data":"930aecc963554c8d58b24a976ee5b0a0044847a4561cd6ef79d572dbc1772205"}
Oct 30 12:15:17.527741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.527717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerStarted","Data":"f9e9a74459ce96699fd839b4cb2e621fd38121dcbfe34f6503a684aadba819dc"}
Oct 30 12:15:17.529650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.529627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerStarted","Data":"c3d03681595070a876f88811db5fdceda0bef1039438fb80176696d722052369"}
Oct 30 12:15:17.531516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.531495    5060 generic.go:356] "Generic (PLEG): container finished" podID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerID="cb3982320c6da0dafcb90a7375ff2dfdab7a1a558f5c76776f0d5d5018b34acb" exitCode=60
Oct 30 12:15:17.531651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.531566    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" event={"ID":"9e0145c1-1650-4d30-a144-81bde1ed9393","Type":"ContainerDied","Data":"cb3982320c6da0dafcb90a7375ff2dfdab7a1a558f5c76776f0d5d5018b34acb"}
Oct 30 12:15:17.531651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.531627    5060 scope.go:122] "RemoveContainer" containerID="8908cfa5631eebc366924cc85b977b171e7e38534d2be8e8df5a62fdaa5b736c"
Oct 30 12:15:17.531972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.531928    5060 scope.go:122] "RemoveContainer" containerID="cb3982320c6da0dafcb90a7375ff2dfdab7a1a558f5c76776f0d5d5018b34acb"
Oct 30 12:15:17.532263 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:17.532237    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql_odh-ai-gateway-infra(9e0145c1-1650-4d30-a144-81bde1ed9393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393"
Oct 30 12:15:17.566412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.566384    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"]
Oct 30 12:15:17.702316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.701557    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" podStartSLOduration=17.70153773 podStartE2EDuration="17.70153773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:17.65792357 +0000 UTC m=+2721.452808065" watchObservedRunningTime="2026-10-30 12:15:17.70153773 +0000 UTC m=+2721.496422228"
Oct 30 12:15:17.900045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:17.900012    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:18.030962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.029792    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca\") pod \"c83881d6-593b-4926-b2f7-601ef59b98e2\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") "
Oct 30 12:15:18.030962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.029950    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2\" (UniqueName: \"kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2\") pod \"c83881d6-593b-4926-b2f7-601ef59b98e2\" (UID: \"c83881d6-593b-4926-b2f7-601ef59b98e2\") "
Oct 30 12:15:18.031643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.031611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca" pod "c83881d6-593b-4926-b2f7-601ef59b98e2" (UID: "c83881d6-593b-4926-b2f7-601ef59b98e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:18.042696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.042662    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2" pod "c83881d6-593b-4926-b2f7-601ef59b98e2" (UID: "c83881d6-593b-4926-b2f7-601ef59b98e2"). InnerVolumeSpecName "kube-api-access-4tsc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:18.131191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.131107    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83881d6-593b-4926-b2f7-601ef59b98e2-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:18.131191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.131134    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tsc2\" (UniqueName: \"kubernetes.io/projected/c83881d6-593b-4926-b2f7-601ef59b98e2-kube-api-access-4tsc2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:18.327442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.327398    5060 scope.go:122] "RemoveContainer" containerID="92f156b8842cfae503700b4f91b838cd8ee143c4aced2b731c65cc6ce2ece5b3"
Oct 30 12:15:18.327664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.327567    5060 scope.go:122] "RemoveContainer" containerID="9e62a3b9f0666303323d23937c49e27022579fc34d3cb47525f3929ca5bc5acf"
Oct 30 12:15:18.327748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.327722    5060 scope.go:122] "RemoveContainer" containerID="5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5"
Oct 30 12:15:18.327857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.327841    5060 scope.go:122] "RemoveContainer" containerID="2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60"
Oct 30 12:15:18.556528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.556492    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerStarted","Data":"fdaaed52e10785ed08d4b03c91c169b42046ad937b1ae0cfbaf9476733bcbcac"}
Oct 30 12:15:18.558432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.558406    5060 generic.go:356] "Generic (PLEG): container finished" podID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerID="4f4affb0011d15f6f871ad47180a9cf42b14084a3405dde6fb244f41863f00d8" exitCode=60
Oct 30 12:15:18.558594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.558481    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" event={"ID":"27208e5c-eac7-4c18-b9e8-b68030349fb9","Type":"ContainerDied","Data":"4f4affb0011d15f6f871ad47180a9cf42b14084a3405dde6fb244f41863f00d8"}
Oct 30 12:15:18.558594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.558534    5060 scope.go:122] "RemoveContainer" containerID="b4dc620bc821078208a60606c22f46753ab8cdb260d1b3dfa325e1002380a955"
Oct 30 12:15:18.558831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.558818    5060 scope.go:122] "RemoveContainer" containerID="4f4affb0011d15f6f871ad47180a9cf42b14084a3405dde6fb244f41863f00d8"
Oct 30 12:15:18.559121 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:18.559099    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6_odh-ai-gateway-infra(27208e5c-eac7-4c18-b9e8-b68030349fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9"
Oct 30 12:15:18.560516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.560499    5060 generic.go:356] "Generic (PLEG): container finished" podID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerID="c3d03681595070a876f88811db5fdceda0bef1039438fb80176696d722052369" exitCode=60
Oct 30 12:15:18.560612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.560572    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerDied","Data":"c3d03681595070a876f88811db5fdceda0bef1039438fb80176696d722052369"}
Oct 30 12:15:18.560944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.560928    5060 scope.go:122] "RemoveContainer" containerID="c3d03681595070a876f88811db5fdceda0bef1039438fb80176696d722052369"
Oct 30 12:15:18.561282 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:18.561257    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx_odh-ai-gateway-infra(a70afb93-c0d8-430b-8d25-8cf630003fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5"
Oct 30 12:15:18.566336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.564053    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp" event={"ID":"c83881d6-593b-4926-b2f7-601ef59b98e2","Type":"ContainerDied","Data":"c6fbf5e30e32ad86421b9207a5a3aa38a43470eb80aa92dc9471e5355a79f4e8"}
Oct 30 12:15:18.566336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.564061    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"
Oct 30 12:15:18.567140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.567114    5060 generic.go:356] "Generic (PLEG): container finished" podID="0b19a738-7614-430a-98bc-30bed2518e8a" containerID="930aecc963554c8d58b24a976ee5b0a0044847a4561cd6ef79d572dbc1772205" exitCode=60
Oct 30 12:15:18.567247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.567231    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerDied","Data":"930aecc963554c8d58b24a976ee5b0a0044847a4561cd6ef79d572dbc1772205"}
Oct 30 12:15:18.567591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.567575    5060 scope.go:122] "RemoveContainer" containerID="930aecc963554c8d58b24a976ee5b0a0044847a4561cd6ef79d572dbc1772205"
Oct 30 12:15:18.567867 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:18.567847    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9_odh-ai-gateway-infra(0b19a738-7614-430a-98bc-30bed2518e8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" podUID="0b19a738-7614-430a-98bc-30bed2518e8a"
Oct 30 12:15:18.572542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.570968    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerStarted","Data":"de5d35c7ac3f7975f13905dc449cae016840cee5e63a211305023b934c51f33b"}
Oct 30 12:15:18.574207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.574186    5060 generic.go:356] "Generic (PLEG): container finished" podID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerID="f9e9a74459ce96699fd839b4cb2e621fd38121dcbfe34f6503a684aadba819dc" exitCode=60
Oct 30 12:15:18.574302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.574260    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerDied","Data":"f9e9a74459ce96699fd839b4cb2e621fd38121dcbfe34f6503a684aadba819dc"}
Oct 30 12:15:18.574575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.574561    5060 scope.go:122] "RemoveContainer" containerID="f9e9a74459ce96699fd839b4cb2e621fd38121dcbfe34f6503a684aadba819dc"
Oct 30 12:15:18.574783 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:18.574767    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g_odh-ai-gateway-infra(6e3fe01a-cd60-4ed4-9a0c-4981a216fc25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25"
Oct 30 12:15:18.586038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.586008    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"]
Oct 30 12:15:18.624278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.624246    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"]
Oct 30 12:15:18.657752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.657709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:18.664570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.664519    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"]
Oct 30 12:15:18.711824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.711791    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"]
Oct 30 12:15:18.875671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.872372    5060 scope.go:122] "RemoveContainer" containerID="e5d7079c5c9bd7e55ffd7f4e719c5ba68f1162238bba14a620dd5dfdee19cbde"
Oct 30 12:15:18.893388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.893334    5060 scope.go:122] "RemoveContainer" containerID="a84f93d07b0c754941db58b8f4c5aeca07f7c03931536043a20579267e18c62e"
Oct 30 12:15:18.906316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.906295    5060 scope.go:122] "RemoveContainer" containerID="5c1d3c1376342016b0ab2cb2179e0324e9551660c4082d3a53b4caee56595db3"
Oct 30 12:15:18.914776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.914695    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"]
Oct 30 12:15:18.922649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.922619    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889375-82qqp"]
Oct 30 12:15:18.923133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:18.923112    5060 scope.go:122] "RemoveContainer" containerID="5b735116a12d2acac5bdbf37f514c2f1bdccf5bce28a6ac04f12ca86f98fdefb"
Oct 30 12:15:19.101497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.101467    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:19.230698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.155059    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm\" (UniqueName: \"kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm\") pod \"9e0145c1-1650-4d30-a144-81bde1ed9393\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") "
Oct 30 12:15:19.230698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.155162    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca\") pod \"9e0145c1-1650-4d30-a144-81bde1ed9393\" (UID: \"9e0145c1-1650-4d30-a144-81bde1ed9393\") "
Oct 30 12:15:19.230698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.161711    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca" pod "9e0145c1-1650-4d30-a144-81bde1ed9393" (UID: "9e0145c1-1650-4d30-a144-81bde1ed9393"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:19.230698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.196670    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm" pod "9e0145c1-1650-4d30-a144-81bde1ed9393" (UID: "9e0145c1-1650-4d30-a144-81bde1ed9393"). InnerVolumeSpecName "kube-api-access-f45xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:19.257359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.257318    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f45xm\" (UniqueName: \"kubernetes.io/projected/9e0145c1-1650-4d30-a144-81bde1ed9393-kube-api-access-f45xm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:19.257642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.257620    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0145c1-1650-4d30-a144-81bde1ed9393-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:19.290338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.290298    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:19.327390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.327354    5060 scope.go:122] "RemoveContainer" containerID="309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c"
Oct 30 12:15:19.327597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.327535    5060 scope.go:122] "RemoveContainer" containerID="76f91a5ab17a1dc95d08324477546918761cdfd24867ce955f0880a525f7fdce"
Oct 30 12:15:19.439815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.439776    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:19.649510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.649476    5060 generic.go:356] "Generic (PLEG): container finished" podID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerID="de5d35c7ac3f7975f13905dc449cae016840cee5e63a211305023b934c51f33b" exitCode=60
Oct 30 12:15:19.649890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.649618    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerDied","Data":"de5d35c7ac3f7975f13905dc449cae016840cee5e63a211305023b934c51f33b"}
Oct 30 12:15:19.649890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.649683    5060 scope.go:122] "RemoveContainer" containerID="19f13700c41462da43e1d43fea1a54c09e3304858e6ee25e3c7de6c7e4b0005a"
Oct 30 12:15:19.653492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.653472    5060 scope.go:122] "RemoveContainer" containerID="de5d35c7ac3f7975f13905dc449cae016840cee5e63a211305023b934c51f33b"
Oct 30 12:15:19.654118 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:19.654091    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns_odh-ai-gateway-infra(e38aaaa9-f3bf-42db-8e40-47761f7b3c3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a"
Oct 30 12:15:19.654266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.654158    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerStarted","Data":"f136bd18cafdde0ff9bc2dbb10e62a3f9d8d6b24efe4ed1c026c88fde356e107"}
Oct 30 12:15:19.661009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.660179    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"]
Oct 30 12:15:19.661009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.660239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerStarted","Data":"7583344e2b25cb6881cc45ccdd497ae16783c1dd752de76b22e6b8438cd55fdb"}
Oct 30 12:15:19.663696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.663429    5060 generic.go:356] "Generic (PLEG): container finished" podID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerID="fdaaed52e10785ed08d4b03c91c169b42046ad937b1ae0cfbaf9476733bcbcac" exitCode=60
Oct 30 12:15:19.663696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.663541    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerDied","Data":"fdaaed52e10785ed08d4b03c91c169b42046ad937b1ae0cfbaf9476733bcbcac"}
Oct 30 12:15:19.667064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.666524    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" event={"ID":"b02e49c9-34d8-44f2-90a3-f4cf680db99b","Type":"ContainerStarted","Data":"9f9cd4d8bdeb13a05e9b08fa75559c0dfc1f9542ea5233fd8f954fbc6b649465"}
Oct 30 12:15:19.682787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.675569    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql" event={"ID":"9e0145c1-1650-4d30-a144-81bde1ed9393","Type":"ContainerDied","Data":"906b055960b042d993b9c9ef5c444eed45ea7911786dc55e3baa7144119b0b32"}
Oct 30 12:15:19.682787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.675667    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"
Oct 30 12:15:19.682787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.682677    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"]
Oct 30 12:15:19.688504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.687287    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerStarted","Data":"1e42cafd1f0d89c4e9a7378e0aa25642ece3d23fc8e3e3736099b0835ab4a86b"}
Oct 30 12:15:19.803698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.795962    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:19.841871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.840982    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"]
Oct 30 12:15:19.928482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:19.928292    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" podStartSLOduration=19.928256059 podStartE2EDuration="19.928256059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:19.841563661 +0000 UTC m=+2723.636448145" watchObservedRunningTime="2026-10-30 12:15:19.928256059 +0000 UTC m=+2723.723140557"
Oct 30 12:15:20.087972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.012396    5060 scope.go:122] "RemoveContainer" containerID="454326b563092ee3cdeaec3c325d016fe003bda757085ad048cae7d3f66a375e"
Oct 30 12:15:20.128370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.105712    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:20.143701 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:20.143503    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:15:20.261365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.256737    5060 scope.go:122] "RemoveContainer" containerID="cb3982320c6da0dafcb90a7375ff2dfdab7a1a558f5c76776f0d5d5018b34acb"
Oct 30 12:15:20.306267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.306242    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:20.317570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.317501    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" podStartSLOduration=20.317481837 podStartE2EDuration="20.317481837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:20.072092973 +0000 UTC m=+2723.866977468" watchObservedRunningTime="2026-10-30 12:15:20.317481837 +0000 UTC m=+2724.112366330"
Oct 30 12:15:20.402869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.377213    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" path="/var/lib/kubelet/pods/c83881d6-593b-4926-b2f7-601ef59b98e2/volumes"
Oct 30 12:15:20.402869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.394961    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"]
Oct 30 12:15:20.430328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.430292    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889375-clsql"]
Oct 30 12:15:20.437122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.437090    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk\" (UniqueName: \"kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk\") pod \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") "
Oct 30 12:15:20.437428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.437410    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca\") pod \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\" (UID: \"5b8cbfdb-36d8-4f32-80ae-ad130c47817e\") "
Oct 30 12:15:20.438851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.438828    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca" pod "5b8cbfdb-36d8-4f32-80ae-ad130c47817e" (UID: "5b8cbfdb-36d8-4f32-80ae-ad130c47817e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:20.448318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.447056    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk" pod "5b8cbfdb-36d8-4f32-80ae-ad130c47817e" (UID: "5b8cbfdb-36d8-4f32-80ae-ad130c47817e"). InnerVolumeSpecName "kube-api-access-tj6kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:20.533508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.530646    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:20.539987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.539955    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj6kk\" (UniqueName: \"kubernetes.io/projected/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-kube-api-access-tj6kk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.540192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.540175    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8cbfdb-36d8-4f32-80ae-ad130c47817e-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.553988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.553302    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:20.651997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.651962    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca\") pod \"a70afb93-c0d8-430b-8d25-8cf630003fb5\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") "
Oct 30 12:15:20.652788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.652730    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca\") pod \"0b19a738-7614-430a-98bc-30bed2518e8a\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") "
Oct 30 12:15:20.652963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.652946    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6\" (UniqueName: \"kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6\") pod \"0b19a738-7614-430a-98bc-30bed2518e8a\" (UID: \"0b19a738-7614-430a-98bc-30bed2518e8a\") "
Oct 30 12:15:20.653560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.653076    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv\" (UniqueName: \"kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv\") pod \"a70afb93-c0d8-430b-8d25-8cf630003fb5\" (UID: \"a70afb93-c0d8-430b-8d25-8cf630003fb5\") "
Oct 30 12:15:20.671524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.669946    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca" pod "a70afb93-c0d8-430b-8d25-8cf630003fb5" (UID: "a70afb93-c0d8-430b-8d25-8cf630003fb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:20.674679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.673313    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca" pod "0b19a738-7614-430a-98bc-30bed2518e8a" (UID: "0b19a738-7614-430a-98bc-30bed2518e8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:20.674679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.674125    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:20.674679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.674296    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv" pod "a70afb93-c0d8-430b-8d25-8cf630003fb5" (UID: "a70afb93-c0d8-430b-8d25-8cf630003fb5"). InnerVolumeSpecName "kube-api-access-cwjtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:20.687861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.687469    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:20.712168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.712111    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6" pod "0b19a738-7614-430a-98bc-30bed2518e8a" (UID: "0b19a738-7614-430a-98bc-30bed2518e8a"). InnerVolumeSpecName "kube-api-access-xlww6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:20.735501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.735378    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:20.737618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.736174    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" event={"ID":"88115e62-3f73-48de-bed3-1c5807fcee69","Type":"ContainerStarted","Data":"a0e3d6bb5c0bb1c9f81e5062edd17ff3545d2cdbc813222fb2fa23f02885b157"}
Oct 30 12:15:20.742717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.742640    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9" event={"ID":"0b19a738-7614-430a-98bc-30bed2518e8a","Type":"ContainerDied","Data":"f3f855d40540c5bbacf36538c4c0042d3c062e3186e8eec1c0d001615a1786bf"}
Oct 30 12:15:20.743007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.742991    5060 scope.go:122] "RemoveContainer" containerID="930aecc963554c8d58b24a976ee5b0a0044847a4561cd6ef79d572dbc1772205"
Oct 30 12:15:20.743495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.743478    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"
Oct 30 12:15:20.755676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.755644    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca\") pod \"27208e5c-eac7-4c18-b9e8-b68030349fb9\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") "
Oct 30 12:15:20.763330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.756728    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca" pod "27208e5c-eac7-4c18-b9e8-b68030349fb9" (UID: "27208e5c-eac7-4c18-b9e8-b68030349fb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:20.763506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.763320    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr\" (UniqueName: \"kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr\") pod \"27208e5c-eac7-4c18-b9e8-b68030349fb9\" (UID: \"27208e5c-eac7-4c18-b9e8-b68030349fb9\") "
Oct 30 12:15:20.764257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.764234    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19a738-7614-430a-98bc-30bed2518e8a-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.764387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.764374    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlww6\" (UniqueName: \"kubernetes.io/projected/0b19a738-7614-430a-98bc-30bed2518e8a-kube-api-access-xlww6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.764521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.764507    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjtv\" (UniqueName: \"kubernetes.io/projected/a70afb93-c0d8-430b-8d25-8cf630003fb5-kube-api-access-cwjtv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.764649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.764637    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27208e5c-eac7-4c18-b9e8-b68030349fb9-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.764740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.764730    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70afb93-c0d8-430b-8d25-8cf630003fb5-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.774595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.774563    5060 generic.go:356] "Generic (PLEG): container finished" podID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerID="f136bd18cafdde0ff9bc2dbb10e62a3f9d8d6b24efe4ed1c026c88fde356e107" exitCode=60
Oct 30 12:15:20.774734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.774714    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerDied","Data":"f136bd18cafdde0ff9bc2dbb10e62a3f9d8d6b24efe4ed1c026c88fde356e107"}
Oct 30 12:15:20.775239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.775218    5060 scope.go:122] "RemoveContainer" containerID="f136bd18cafdde0ff9bc2dbb10e62a3f9d8d6b24efe4ed1c026c88fde356e107"
Oct 30 12:15:20.794427 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:20.794386    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws_odh-ai-gateway-infra(7ea959e0-fac6-4c6c-96c1-5e106c45e3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9"
Oct 30 12:15:20.838186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.837021    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr" pod "27208e5c-eac7-4c18-b9e8-b68030349fb9" (UID: "27208e5c-eac7-4c18-b9e8-b68030349fb9"). InnerVolumeSpecName "kube-api-access-zn4xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:20.838578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.838556    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"
Oct 30 12:15:20.838880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.838566    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g" event={"ID":"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25","Type":"ContainerDied","Data":"6d5d84a9a87e82afddc08a79cef32c46071c9fdcfa0050dd41196ad46880f803"}
Oct 30 12:15:20.930036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.900176    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca\") pod \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") "
Oct 30 12:15:20.930036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.900299    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw\" (UniqueName: \"kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw\") pod \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\" (UID: \"6e3fe01a-cd60-4ed4-9a0c-4981a216fc25\") "
Oct 30 12:15:20.930036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.901106    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4xr\" (UniqueName: \"kubernetes.io/projected/27208e5c-eac7-4c18-b9e8-b68030349fb9-kube-api-access-zn4xr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:20.930036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.901639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca" pod "6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" (UID: "6e3fe01a-cd60-4ed4-9a0c-4981a216fc25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:20.944701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.944666    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw" pod "6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" (UID: "6e3fe01a-cd60-4ed4-9a0c-4981a216fc25"). InnerVolumeSpecName "kube-api-access-lcqbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:20.951489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.947373    5060 scope.go:122] "RemoveContainer" containerID="5624d180c2a5cbef98aaeaa15dfceb52834c8ce7a20374db3856868c698ca3f5"
Oct 30 12:15:20.951489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.947734    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:20.982349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.979244    5060 generic.go:356] "Generic (PLEG): container finished" podID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerID="7583344e2b25cb6881cc45ccdd497ae16783c1dd752de76b22e6b8438cd55fdb" exitCode=60
Oct 30 12:15:20.982349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.979608    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerDied","Data":"7583344e2b25cb6881cc45ccdd497ae16783c1dd752de76b22e6b8438cd55fdb"}
Oct 30 12:15:20.982349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.981885    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"]
Oct 30 12:15:20.990766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.989872    5060 scope.go:122] "RemoveContainer" containerID="7583344e2b25cb6881cc45ccdd497ae16783c1dd752de76b22e6b8438cd55fdb"
Oct 30 12:15:20.990766 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:20.990291    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z_odh-ai-gateway-infra(b511f39e-f4b2-494e-86ad-9c4a66b8920b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b"
Oct 30 12:15:20.999356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:20.998792    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"]
Oct 30 12:15:21.039285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.025928    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"]
Oct 30 12:15:21.045644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.041034    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerStarted","Data":"05ae51ec7c40e84c6dd59689e232ef652e3f08fb4080719ae59a8d6547124209"}
Oct 30 12:15:21.045644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.041490    5060 scope.go:122] "RemoveContainer" containerID="05ae51ec7c40e84c6dd59689e232ef652e3f08fb4080719ae59a8d6547124209"
Oct 30 12:15:21.045644 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:21.041843    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889375-stmvk_odh-ai-gateway-infra(6813457a-74d2-4a5c-8b96-0e0fa0909751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751"
Oct 30 12:15:21.045644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.043162    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcqbw\" (UniqueName: \"kubernetes.io/projected/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-kube-api-access-lcqbw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.045644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.043187    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.056535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.056496    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2" event={"ID":"5b8cbfdb-36d8-4f32-80ae-ad130c47817e","Type":"ContainerDied","Data":"bf37b569ffdff2f74da03c8c3a0d8082c8097164c07b97b38d40de6f1e2697f4"}
Oct 30 12:15:21.056760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.056512    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"
Oct 30 12:15:21.062407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.062374    5060 generic.go:356] "Generic (PLEG): container finished" podID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerID="9f9cd4d8bdeb13a05e9b08fa75559c0dfc1f9542ea5233fd8f954fbc6b649465" exitCode=60
Oct 30 12:15:21.062685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.062663    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" event={"ID":"b02e49c9-34d8-44f2-90a3-f4cf680db99b","Type":"ContainerDied","Data":"9f9cd4d8bdeb13a05e9b08fa75559c0dfc1f9542ea5233fd8f954fbc6b649465"}
Oct 30 12:15:21.063302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.063284    5060 scope.go:122] "RemoveContainer" containerID="9f9cd4d8bdeb13a05e9b08fa75559c0dfc1f9542ea5233fd8f954fbc6b649465"
Oct 30 12:15:21.063806 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:21.063785    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc_odh-ai-gateway-infra(b02e49c9-34d8-44f2-90a3-f4cf680db99b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b"
Oct 30 12:15:21.066959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.066858    5060 scope.go:122] "RemoveContainer" containerID="f9e9a74459ce96699fd839b4cb2e621fd38121dcbfe34f6503a684aadba819dc"
Oct 30 12:15:21.069900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.068332    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6" event={"ID":"27208e5c-eac7-4c18-b9e8-b68030349fb9","Type":"ContainerDied","Data":"d8d0fc3f8dbdb345eb401c199b295b9018721e2315f160d83cf7c774ab6a7f7b"}
Oct 30 12:15:21.069900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.068424    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"
Oct 30 12:15:21.073923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.073886    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"
Oct 30 12:15:21.074058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.073886    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx" event={"ID":"a70afb93-c0d8-430b-8d25-8cf630003fb5","Type":"ContainerDied","Data":"ea2ba5d8526e8e0d62fabfe450f5581c87fb4add0de8cfb9935ae166cc71775f"}
Oct 30 12:15:21.134791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.134761    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"]
Oct 30 12:15:21.151201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.151173    5060 scope.go:122] "RemoveContainer" containerID="2ffa36e674a1b8a438e77d90423fd4e78c2431d03dd38e46501bcd02eff43b60"
Oct 30 12:15:21.155857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.155828    5060 generic.go:356] "Generic (PLEG): container finished" podID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerID="1e42cafd1f0d89c4e9a7378e0aa25642ece3d23fc8e3e3736099b0835ab4a86b" exitCode=60
Oct 30 12:15:21.156117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.156083    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerDied","Data":"1e42cafd1f0d89c4e9a7378e0aa25642ece3d23fc8e3e3736099b0835ab4a86b"}
Oct 30 12:15:21.278874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.277555    5060 scope.go:122] "RemoveContainer" containerID="fdaaed52e10785ed08d4b03c91c169b42046ad937b1ae0cfbaf9476733bcbcac"
Oct 30 12:15:21.311198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.311153    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"]
Oct 30 12:15:21.325476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.322834    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889375-tsjg2"]
Oct 30 12:15:21.355716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.355680    5060 scope.go:122] "RemoveContainer" containerID="92f156b8842cfae503700b4f91b838cd8ee143c4aced2b731c65cc6ce2ece5b3"
Oct 30 12:15:21.446616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.446588    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:21.454134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.449498    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"]
Oct 30 12:15:21.458092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.458062    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889375-74jg9"]
Oct 30 12:15:21.469483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.469438    5060 scope.go:122] "RemoveContainer" containerID="4f4affb0011d15f6f871ad47180a9cf42b14084a3405dde6fb244f41863f00d8"
Oct 30 12:15:21.540658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.540631    5060 scope.go:122] "RemoveContainer" containerID="c3d03681595070a876f88811db5fdceda0bef1039438fb80176696d722052369"
Oct 30 12:15:21.552436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.548097    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"]
Oct 30 12:15:21.552745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.552728    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889375-xp7w6"]
Oct 30 12:15:21.568610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.568518    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"]
Oct 30 12:15:21.572650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.572626    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889375-xs44g"]
Oct 30 12:15:21.625273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.623892    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5\") pod \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") "
Oct 30 12:15:21.625273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.624058    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca\") pod \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\" (UID: \"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a\") "
Oct 30 12:15:21.625669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.625601    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"]
Oct 30 12:15:21.625964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.625922    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca" pod "e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" (UID: "e38aaaa9-f3bf-42db-8e40-47761f7b3c3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:21.632242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.632191    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5" pod "e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" (UID: "e38aaaa9-f3bf-42db-8e40-47761f7b3c3a"). InnerVolumeSpecName "kube-api-access-lgzl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:21.633645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.633488    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889375-pljvx"]
Oct 30 12:15:21.639747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.639722    5060 scope.go:122] "RemoveContainer" containerID="9e62a3b9f0666303323d23937c49e27022579fc34d3cb47525f3929ca5bc5acf"
Oct 30 12:15:21.669300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.669264    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:21.726728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.726435    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.726728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.726480    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a-kube-api-access-lgzl5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.831668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.831564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca\") pod \"600d3b71-9d75-4bf8-8e63-d54049d6334e\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") "
Oct 30 12:15:21.831856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.831765    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w\" (UniqueName: \"kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w\") pod \"600d3b71-9d75-4bf8-8e63-d54049d6334e\" (UID: \"600d3b71-9d75-4bf8-8e63-d54049d6334e\") "
Oct 30 12:15:21.832713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.832689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca" pod "600d3b71-9d75-4bf8-8e63-d54049d6334e" (UID: "600d3b71-9d75-4bf8-8e63-d54049d6334e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:21.835625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.835601    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w" pod "600d3b71-9d75-4bf8-8e63-d54049d6334e" (UID: "600d3b71-9d75-4bf8-8e63-d54049d6334e"). InnerVolumeSpecName "kube-api-access-tqv6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:21.913013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.912968    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"]
Oct 30 12:15:21.926435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.926395    5060 cpu_manager.go:414] "containerMap: removing container" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.926677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.926661    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.926771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.926761    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:21.926848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.926837    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:21.926964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.926953    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.927051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927040    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.927134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927125    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.927207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927197    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.927289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927279    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e585830e-175e-475c-8598-ce9b4ef4f9d5" containerName="payload-processing"
Oct 30 12:15:21.927361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927352    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e585830e-175e-475c-8598-ce9b4ef4f9d5" containerName="payload-processing"
Oct 30 12:15:21.927471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927442    5060 cpu_manager.go:414] "containerMap: removing container" podUID="24509a9d-77c1-4139-8f59-c534fd7d1691" containerName="cleanup"
Oct 30 12:15:21.927578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927567    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24509a9d-77c1-4139-8f59-c534fd7d1691" containerName="cleanup"
Oct 30 12:15:21.927657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927648    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.927729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927719    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.927805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927796    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.927884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927875    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.927966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.927957    5060 cpu_manager.go:414] "containerMap: removing container" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.928042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928033    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.928121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928113    5060 cpu_manager.go:414] "containerMap: removing container" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.928198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928189    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.928278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928269    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.928351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928342    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.928434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928422    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.928612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928598    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.928709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928700    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.928793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928783    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.928888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928879    5060 cpu_manager.go:414] "containerMap: removing container" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.928964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.928954    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.929045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929033    5060 cpu_manager.go:414] "containerMap: removing container" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.929126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929116    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.929205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929196    5060 cpu_manager.go:414] "containerMap: removing container" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.929277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929268    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.929360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929351    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.929434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929424    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.929538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929527    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.929633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929622    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.929714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929705    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.929793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929783    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.929890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929880    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.929973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.929963    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.930051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.930042    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.930133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.930122    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.930236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.930224    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.931916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.930307    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.932082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.932062    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.932233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.932217    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.932329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.932318    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:21.932533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.932517    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:21.932661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.932648    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.932751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.932740    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.933066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.933046    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.933185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.933175    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.934649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.934630    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.934785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.934771    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:21.934883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.934873    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.934990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.934979    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.935101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935090    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.935183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935174    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.935265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935255    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.935346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935337    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.935446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935435    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.935568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935557    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" containerName="cleanup"
Oct 30 12:15:21.935643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935634    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:21.935735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935726    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" containerName="cleanup"
Oct 30 12:15:21.935816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935808    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:21.935904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935895    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.935987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.935978    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.936089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936078    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e585830e-175e-475c-8598-ce9b4ef4f9d5" containerName="payload-processing"
Oct 30 12:15:21.936184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936175    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="24509a9d-77c1-4139-8f59-c534fd7d1691" containerName="cleanup"
Oct 30 12:15:21.936261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936252    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:21.936343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936331    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:21.936421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936412    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.936547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936534    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:21.936668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936656    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:21.936751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.936742    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:21.939189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.934299    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqv6w\" (UniqueName: \"kubernetes.io/projected/600d3b71-9d75-4bf8-8e63-d54049d6334e-kube-api-access-tqv6w\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.939338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.939322    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600d3b71-9d75-4bf8-8e63-d54049d6334e-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.955793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.955761    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"]
Oct 30 12:15:21.956153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.956081    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:21.962740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.960015    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"]
Oct 30 12:15:21.973058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:21.973030    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-67e819-openshift-default-dockercfg-srp4d\""
Oct 30 12:15:22.064481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.035032    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073248    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073409    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073470    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073556    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073665    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073700    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.147523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.073808    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqz95\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.175319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.175273    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.175670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.175645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.181576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.181547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.181901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.181879    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqz95\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.182017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.182005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.182114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.182103    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.182292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.182281    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.182401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.182390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.182597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.182583    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.184903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.180902    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.185572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.185539    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.186291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.186266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.187043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.187021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.206526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.192814    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.206526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.197291    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.237534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.220900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.237534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.221427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:22.237534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.224137    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqz95\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.275271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.267831    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token\") pod \"e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") " pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.275271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.274664    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:22.365757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.365729    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"
Oct 30 12:15:22.443805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.443767    5060 generic.go:356] "Generic (PLEG): container finished" podID="88115e62-3f73-48de-bed3-1c5807fcee69" containerID="a0e3d6bb5c0bb1c9f81e5062edd17ff3545d2cdbc813222fb2fa23f02885b157" exitCode=60
Oct 30 12:15:22.444031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.444001    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" path="/var/lib/kubelet/pods/0b19a738-7614-430a-98bc-30bed2518e8a/volumes"
Oct 30 12:15:22.444886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.444864    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" path="/var/lib/kubelet/pods/27208e5c-eac7-4c18-b9e8-b68030349fb9/volumes"
Oct 30 12:15:22.446160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.446137    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8cbfdb-36d8-4f32-80ae-ad130c47817e" path="/var/lib/kubelet/pods/5b8cbfdb-36d8-4f32-80ae-ad130c47817e/volumes"
Oct 30 12:15:22.447173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.447151    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3fe01a-cd60-4ed4-9a0c-4981a216fc25" path="/var/lib/kubelet/pods/6e3fe01a-cd60-4ed4-9a0c-4981a216fc25/volumes"
Oct 30 12:15:22.447727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.447706    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" path="/var/lib/kubelet/pods/9e0145c1-1650-4d30-a144-81bde1ed9393/volumes"
Oct 30 12:15:22.448237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.448217    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" path="/var/lib/kubelet/pods/a70afb93-c0d8-430b-8d25-8cf630003fb5/volumes"
Oct 30 12:15:22.449596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.449574    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4" event={"ID":"600d3b71-9d75-4bf8-8e63-d54049d6334e","Type":"ContainerDied","Data":"c88514cc0a079353295a6887f2e09ef832c11e4963b9e508dcd7f38b0ce234e6"}
Oct 30 12:15:22.449740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.449725    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"]
Oct 30 12:15:22.449871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.449857    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" event={"ID":"88115e62-3f73-48de-bed3-1c5807fcee69","Type":"ContainerDied","Data":"a0e3d6bb5c0bb1c9f81e5062edd17ff3545d2cdbc813222fb2fa23f02885b157"}
Oct 30 12:15:22.459799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.459190    5060 scope.go:122] "RemoveContainer" containerID="1e42cafd1f0d89c4e9a7378e0aa25642ece3d23fc8e3e3736099b0835ab4a86b"
Oct 30 12:15:22.499587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.488785    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns" event={"ID":"e38aaaa9-f3bf-42db-8e40-47761f7b3c3a","Type":"ContainerDied","Data":"4db166aaf89a16cb810d9b4885daab306d34b204505045f8e951c2cabf3b477d"}
Oct 30 12:15:22.499587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.488908    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"
Oct 30 12:15:22.541083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.541050    5060 generic.go:356] "Generic (PLEG): container finished" podID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerID="05ae51ec7c40e84c6dd59689e232ef652e3f08fb4080719ae59a8d6547124209" exitCode=60
Oct 30 12:15:22.541316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.541297    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerDied","Data":"05ae51ec7c40e84c6dd59689e232ef652e3f08fb4080719ae59a8d6547124209"}
Oct 30 12:15:22.596039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.596013    5060 scope.go:122] "RemoveContainer" containerID="76f91a5ab17a1dc95d08324477546918761cdfd24867ce955f0880a525f7fdce"
Oct 30 12:15:22.663755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.655560    5060 scope.go:122] "RemoveContainer" containerID="de5d35c7ac3f7975f13905dc449cae016840cee5e63a211305023b934c51f33b"
Oct 30 12:15:22.670179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.669235    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:22.740481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.737696    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"]
Oct 30 12:15:22.832575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.832071    5060 scope.go:122] "RemoveContainer" containerID="309454c5b0aafb238e1934aef683d4513a2ab06f6b738aca6f38ff97077c169c"
Oct 30 12:15:22.938566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.938535    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889375-xf7n4"]
Oct 30 12:15:22.952574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.952547    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"]
Oct 30 12:15:22.960028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.960002    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889375-mzlns"]
Oct 30 12:15:22.964302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:22.964275    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:23.038964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.038926    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz\" (UniqueName: \"kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz\") pod \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") "
Oct 30 12:15:23.039278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.039262    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca\") pod \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\" (UID: \"b02e49c9-34d8-44f2-90a3-f4cf680db99b\") "
Oct 30 12:15:23.040811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.040780    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca" pod "b02e49c9-34d8-44f2-90a3-f4cf680db99b" (UID: "b02e49c9-34d8-44f2-90a3-f4cf680db99b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.043852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.043827    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz" pod "b02e49c9-34d8-44f2-90a3-f4cf680db99b" (UID: "b02e49c9-34d8-44f2-90a3-f4cf680db99b"). InnerVolumeSpecName "kube-api-access-fctsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.143998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.143961    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e49c9-34d8-44f2-90a3-f4cf680db99b-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.143998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.143999    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctsz\" (UniqueName: \"kubernetes.io/projected/b02e49c9-34d8-44f2-90a3-f4cf680db99b-kube-api-access-fctsz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.246689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.246029    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:23.248748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.248675    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:23.256483 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:23.254084    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b033509_5c13_468f_8c29_28ff246b975f.slice/crio-f93bce09bc8b8b187f7079be65c48126f13296df5104617fa279ce1cfbd304a0 WatchSource:0}: Error finding container f93bce09bc8b8b187f7079be65c48126f13296df5104617fa279ce1cfbd304a0: Status 404 returned error can't find the container with id f93bce09bc8b8b187f7079be65c48126f13296df5104617fa279ce1cfbd304a0
Oct 30 12:15:23.256483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.255368    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"]
Oct 30 12:15:23.258943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.257285    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:23.258943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.257350    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:23.258943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.257379    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:23.264730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.264610    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:23.268915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.268893    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349505    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca\") pod \"6813457a-74d2-4a5c-8b96-0e0fa0909751\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349540    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca\") pod \"88115e62-3f73-48de-bed3-1c5807fcee69\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84\" (UniqueName: \"kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84\") pod \"88115e62-3f73-48de-bed3-1c5807fcee69\" (UID: \"88115e62-3f73-48de-bed3-1c5807fcee69\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349652    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca\") pod \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349740    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca\") pod \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349765    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7\" (UniqueName: \"kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7\") pod \"6813457a-74d2-4a5c-8b96-0e0fa0909751\" (UID: \"6813457a-74d2-4a5c-8b96-0e0fa0909751\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349797    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm\" (UniqueName: \"kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm\") pod \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\" (UID: \"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9\") "
Oct 30 12:15:23.350639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.349831    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh\" (UniqueName: \"kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh\") pod \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\" (UID: \"b511f39e-f4b2-494e-86ad-9c4a66b8920b\") "
Oct 30 12:15:23.351211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.350744    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca" pod "88115e62-3f73-48de-bed3-1c5807fcee69" (UID: "88115e62-3f73-48de-bed3-1c5807fcee69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.351211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.351086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca" pod "6813457a-74d2-4a5c-8b96-0e0fa0909751" (UID: "6813457a-74d2-4a5c-8b96-0e0fa0909751"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.351544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.351417    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca" pod "b511f39e-f4b2-494e-86ad-9c4a66b8920b" (UID: "b511f39e-f4b2-494e-86ad-9c4a66b8920b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.352161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.352136    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca" pod "7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" (UID: "7ea959e0-fac6-4c6c-96c1-5e106c45e3c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.364298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.359599    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm" pod "7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" (UID: "7ea959e0-fac6-4c6c-96c1-5e106c45e3c9"). InnerVolumeSpecName "kube-api-access-8l8wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.364298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.359652    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84" pod "88115e62-3f73-48de-bed3-1c5807fcee69" (UID: "88115e62-3f73-48de-bed3-1c5807fcee69"). InnerVolumeSpecName "kube-api-access-r4p84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.364298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.360334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh" pod "b511f39e-f4b2-494e-86ad-9c4a66b8920b" (UID: "b511f39e-f4b2-494e-86ad-9c4a66b8920b"). InnerVolumeSpecName "kube-api-access-cl5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.366887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.366266    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7" pod "6813457a-74d2-4a5c-8b96-0e0fa0909751" (UID: "6813457a-74d2-4a5c-8b96-0e0fa0909751"). InnerVolumeSpecName "kube-api-access-vs7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451153    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451188    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b511f39e-f4b2-494e-86ad-9c4a66b8920b-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451203    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7r7\" (UniqueName: \"kubernetes.io/projected/6813457a-74d2-4a5c-8b96-0e0fa0909751-kube-api-access-vs7r7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451218    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8wm\" (UniqueName: \"kubernetes.io/projected/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9-kube-api-access-8l8wm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451230    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl5lh\" (UniqueName: \"kubernetes.io/projected/b511f39e-f4b2-494e-86ad-9c4a66b8920b-kube-api-access-cl5lh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451244    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6813457a-74d2-4a5c-8b96-0e0fa0909751-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451257    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88115e62-3f73-48de-bed3-1c5807fcee69-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.451274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.451269    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4p84\" (UniqueName: \"kubernetes.io/projected/88115e62-3f73-48de-bed3-1c5807fcee69-kube-api-access-r4p84\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.558545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.558049    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"
Oct 30 12:15:23.558545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.558079    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws" event={"ID":"7ea959e0-fac6-4c6c-96c1-5e106c45e3c9","Type":"ContainerDied","Data":"c5c7a711c53e69166c30df5737db618c17be650402affab08d5599f50091d87c"}
Oct 30 12:15:23.558545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.558155    5060 scope.go:122] "RemoveContainer" containerID="f136bd18cafdde0ff9bc2dbb10e62a3f9d8d6b24efe4ed1c026c88fde356e107"
Oct 30 12:15:23.560220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.560188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z" event={"ID":"b511f39e-f4b2-494e-86ad-9c4a66b8920b","Type":"ContainerDied","Data":"8abeffcb23f590ff68a90bf905c2c5a9397552ca3cb50081656488a02b09ab5e"}
Oct 30 12:15:23.560325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.560311    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"
Oct 30 12:15:23.562920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.562902    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"
Oct 30 12:15:23.563960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.562919    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk" event={"ID":"6813457a-74d2-4a5c-8b96-0e0fa0909751","Type":"ContainerDied","Data":"a037dd509807cd71107b5f12ba3a0e018e5e0f74ccecfc5270a45978e6a42ed7"}
Oct 30 12:15:23.564465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.564424    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"
Oct 30 12:15:23.564558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.564534    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc" event={"ID":"b02e49c9-34d8-44f2-90a3-f4cf680db99b","Type":"ContainerDied","Data":"db275485ff3ec7e6185eb8c2934609cffa1b53fafc22c772c7be3e5c25759d7e"}
Oct 30 12:15:23.567471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.567388    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6" event={"ID":"88115e62-3f73-48de-bed3-1c5807fcee69","Type":"ContainerDied","Data":"5e06d41f7c0ea5f31c113d26ff3b37d7fb803b3923dc52866f46c2514fb22b70"}
Oct 30 12:15:23.568533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.567620    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"
Oct 30 12:15:23.612069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.611955    5060 scope.go:122] "RemoveContainer" containerID="7583344e2b25cb6881cc45ccdd497ae16783c1dd752de76b22e6b8438cd55fdb"
Oct 30 12:15:23.614303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.614185    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" event={"ID":"1b033509-5c13-468f-8c29-28ff246b975f","Type":"ContainerStarted","Data":"1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"}
Oct 30 12:15:23.614303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.614246    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" event={"ID":"1b033509-5c13-468f-8c29-28ff246b975f","Type":"ContainerStarted","Data":"f93bce09bc8b8b187f7079be65c48126f13296df5104617fa279ce1cfbd304a0"}
Oct 30 12:15:23.651597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.651577    5060 scope.go:122] "RemoveContainer" containerID="05ae51ec7c40e84c6dd59689e232ef652e3f08fb4080719ae59a8d6547124209"
Oct 30 12:15:23.664405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.664237    5060 scope.go:122] "RemoveContainer" containerID="9f9cd4d8bdeb13a05e9b08fa75559c0dfc1f9542ea5233fd8f954fbc6b649465"
Oct 30 12:15:23.696682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.696650    5060 scope.go:122] "RemoveContainer" containerID="a0e3d6bb5c0bb1c9f81e5062edd17ff3545d2cdbc813222fb2fa23f02885b157"
Oct 30 12:15:23.719795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.719761    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"]
Oct 30 12:15:23.730031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.729773    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889375-qjhx6"]
Oct 30 12:15:23.748654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.748622    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"]
Oct 30 12:15:23.755356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.755327    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889375-mr54z"]
Oct 30 12:15:23.801477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.797510    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"]
Oct 30 12:15:23.812970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.812880    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889375-cvwws"]
Oct 30 12:15:23.846200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.845851    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"]
Oct 30 12:15:23.849446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.849410    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-stmvk"]
Oct 30 12:15:23.919477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.918664    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podStartSLOduration=2.918643112 podStartE2EDuration="2.918643112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:23.897853897 +0000 UTC m=+2727.692738393" watchObservedRunningTime="2026-10-30 12:15:23.918643112 +0000 UTC m=+2727.713527607"
Oct 30 12:15:23.942793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.942713    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"]
Oct 30 12:15:23.946792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:23.946757    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889375-dw2pc"]
Oct 30 12:15:24.275574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.275542    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:24.296479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.295510    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:24.369605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.369569    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" path="/var/lib/kubelet/pods/600d3b71-9d75-4bf8-8e63-d54049d6334e/volumes"
Oct 30 12:15:24.370354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.370330    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" path="/var/lib/kubelet/pods/6813457a-74d2-4a5c-8b96-0e0fa0909751/volumes"
Oct 30 12:15:24.370761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.370745    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" path="/var/lib/kubelet/pods/7ea959e0-fac6-4c6c-96c1-5e106c45e3c9/volumes"
Oct 30 12:15:24.371403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.371387    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" path="/var/lib/kubelet/pods/88115e62-3f73-48de-bed3-1c5807fcee69/volumes"
Oct 30 12:15:24.372048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.372019    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" path="/var/lib/kubelet/pods/b02e49c9-34d8-44f2-90a3-f4cf680db99b/volumes"
Oct 30 12:15:24.372399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.372387    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" path="/var/lib/kubelet/pods/b511f39e-f4b2-494e-86ad-9c4a66b8920b/volumes"
Oct 30 12:15:24.373044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.373029    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" path="/var/lib/kubelet/pods/e38aaaa9-f3bf-42db-8e40-47761f7b3c3a/volumes"
Oct 30 12:15:24.531434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.531344    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:24.649406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.649376    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:24.650531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:24.650514    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:15:26.111379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.111334    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"]
Oct 30 12:15:26.112295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112272    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:26.112295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112296    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112311    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112318    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112338    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112346    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b19a738-7614-430a-98bc-30bed2518e8a" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112368    5060 cpu_manager.go:414] "containerMap: removing container" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112376    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112390    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112396    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112407    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112415    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112432    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112439    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112447    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112472    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0145c1-1650-4d30-a144-81bde1ed9393" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112484    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:26.112499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112491    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38aaaa9-f3bf-42db-8e40-47761f7b3c3a" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112513    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112520    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112535    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112542    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112551    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112559    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112578    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112585    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112612    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112619    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112630    5060 cpu_manager.go:414] "containerMap: removing container" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112637    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112796    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70afb93-c0d8-430b-8d25-8cf630003fb5" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112806    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112815    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112824    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112831    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112840    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112850    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112857    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112867    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83881d6-593b-4926-b2f7-601ef59b98e2" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112878    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112891    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112900    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112909    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="27208e5c-eac7-4c18-b9e8-b68030349fb9" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112920    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d3b71-9d75-4bf8-8e63-d54049d6334e" containerName="cleanup"
Oct 30 12:15:26.113384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.112928    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:26.118729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.118705    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.131862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.131829    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"]
Oct 30 12:15:26.141183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.140813    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:15:26.141183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.140961    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:15:26.141183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.140995    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:15:26.146000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.145975    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:15:26.210874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.210831    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.211055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.211006    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.211055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.211042    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6f84\" (UniqueName: \"kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.312916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.312832    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.313094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.312964    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.313094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.313002    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6f84\" (UniqueName: \"kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.314133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.314104    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.315574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.315551    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.341223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.341190    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6f84\" (UniqueName: \"kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84\") pod \"maas-api-revoke-sub-d7adcf4c-rpjff\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.365218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.365182    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:26.451663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.451632    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:26.524413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.524341    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:26.890143 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:26.890094    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400d405b_a148_4d3d_a0a8_a7c294372c52.slice/crio-fab60c7793dfe7111457c92bac0feb6d6f3ed5ab064e0fb80dedb033ca688674 WatchSource:0}: Error finding container fab60c7793dfe7111457c92bac0feb6d6f3ed5ab064e0fb80dedb033ca688674: Status 404 returned error can't find the container with id fab60c7793dfe7111457c92bac0feb6d6f3ed5ab064e0fb80dedb033ca688674
Oct 30 12:15:26.893603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:26.893431    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"]
Oct 30 12:15:27.690676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:27.690628    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff" event={"ID":"400d405b-a148-4d3d-a0a8-a7c294372c52","Type":"ContainerStarted","Data":"285bcfa4c0eb73548b1dd43e4691277a4907f135252fdee86b7570f3bd6c4580"}
Oct 30 12:15:27.691197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:27.690695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff" event={"ID":"400d405b-a148-4d3d-a0a8-a7c294372c52","Type":"ContainerStarted","Data":"fab60c7793dfe7111457c92bac0feb6d6f3ed5ab064e0fb80dedb033ca688674"}
Oct 30 12:15:27.732793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:27.732725    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff" podStartSLOduration=1.732706361 podStartE2EDuration="1.732706361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:27.731069042 +0000 UTC m=+2731.525953536" watchObservedRunningTime="2026-10-30 12:15:27.732706361 +0000 UTC m=+2731.527590923"
Oct 30 12:15:28.082001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.081812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:28.230541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.230438    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:28.405196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.404710    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.512057    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"]
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.512955    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.512974    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513008    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513017    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513065    5060 cpu_manager.go:414] "containerMap: removing container" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513072    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88115e62-3f73-48de-bed3-1c5807fcee69" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513088    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513095    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513268    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b511f39e-f4b2-494e-86ad-9c4a66b8920b" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513285    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02e49c9-34d8-44f2-90a3-f4cf680db99b" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513295    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea959e0-fac6-4c6c-96c1-5e106c45e3c9" containerName="cleanup"
Oct 30 12:15:28.515330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.513310    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6813457a-74d2-4a5c-8b96-0e0fa0909751" containerName="cleanup"
Oct 30 12:15:28.528642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.528610    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"]
Oct 30 12:15:28.528818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.528679    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.546253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.546228    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-67e819\""
Oct 30 12:15:28.549105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.549077    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-67e819-dockercfg-8ftcz\""
Oct 30 12:15:28.680144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.680096    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.680342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.680165    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfqb\" (UniqueName: \"kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb\") pod \"payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.701211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.701166    5060 generic.go:356] "Generic (PLEG): container finished" podID="400d405b-a148-4d3d-a0a8-a7c294372c52" containerID="285bcfa4c0eb73548b1dd43e4691277a4907f135252fdee86b7570f3bd6c4580" exitCode=0
Oct 30 12:15:28.701643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.701261    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff" event={"ID":"400d405b-a148-4d3d-a0a8-a7c294372c52","Type":"ContainerDied","Data":"285bcfa4c0eb73548b1dd43e4691277a4907f135252fdee86b7570f3bd6c4580"}
Oct 30 12:15:28.781557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.781445    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfqb\" (UniqueName: \"kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb\") pod \"payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.781808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.781779    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.782626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.782602    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.806050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.806014    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfqb\" (UniqueName: \"kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb\") pod \"payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:28.848308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:28.848258    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:29.165844 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:29.165813    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd701a8_818f_456a_8c94_fc10bffa13cc.slice/crio-07c76c248a83fb921a96017799ac2cd6b7238c2dfcd7f67847b5c3aa216debcf WatchSource:0}: Error finding container 07c76c248a83fb921a96017799ac2cd6b7238c2dfcd7f67847b5c3aa216debcf: Status 404 returned error can't find the container with id 07c76c248a83fb921a96017799ac2cd6b7238c2dfcd7f67847b5c3aa216debcf
Oct 30 12:15:29.167493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:29.167431    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"]
Oct 30 12:15:29.734091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:29.734043    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" event={"ID":"edd701a8-818f-456a-8c94-fc10bffa13cc","Type":"ContainerStarted","Data":"eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674"}
Oct 30 12:15:29.734597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:29.734157    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" event={"ID":"edd701a8-818f-456a-8c94-fc10bffa13cc","Type":"ContainerStarted","Data":"07c76c248a83fb921a96017799ac2cd6b7238c2dfcd7f67847b5c3aa216debcf"}
Oct 30 12:15:29.735494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:29.735422    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:29.780382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:29.780031    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" podStartSLOduration=1.780010244 podStartE2EDuration="1.780010244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:29.776570174 +0000 UTC m=+2733.571454670" watchObservedRunningTime="2026-10-30 12:15:29.780010244 +0000 UTC m=+2733.574894739"
Oct 30 12:15:29.963880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:29.963851    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:30.105400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.105311    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca\") pod \"400d405b-a148-4d3d-a0a8-a7c294372c52\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") "
Oct 30 12:15:30.105400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.105387    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84\" (UniqueName: \"kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84\") pod \"400d405b-a148-4d3d-a0a8-a7c294372c52\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") "
Oct 30 12:15:30.105643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.105559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca\") pod \"400d405b-a148-4d3d-a0a8-a7c294372c52\" (UID: \"400d405b-a148-4d3d-a0a8-a7c294372c52\") "
Oct 30 12:15:30.105787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.105760    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca" pod "400d405b-a148-4d3d-a0a8-a7c294372c52" (UID: "400d405b-a148-4d3d-a0a8-a7c294372c52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:30.106162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.106133    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:30.107962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.107928    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84" pod "400d405b-a148-4d3d-a0a8-a7c294372c52" (UID: "400d405b-a148-4d3d-a0a8-a7c294372c52"). InnerVolumeSpecName "kube-api-access-f6f84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:30.108412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.108385    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca" pod "400d405b-a148-4d3d-a0a8-a7c294372c52" (UID: "400d405b-a148-4d3d-a0a8-a7c294372c52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:15:30.207269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.207232    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400d405b-a148-4d3d-a0a8-a7c294372c52-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:30.207430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.207272    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6f84\" (UniqueName: \"kubernetes.io/projected/400d405b-a148-4d3d-a0a8-a7c294372c52-kube-api-access-f6f84\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:15:30.231907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.230541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:30.743498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.743463    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"
Oct 30 12:15:30.743498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.743489    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff" event={"ID":"400d405b-a148-4d3d-a0a8-a7c294372c52","Type":"ContainerDied","Data":"fab60c7793dfe7111457c92bac0feb6d6f3ed5ab064e0fb80dedb033ca688674"}
Oct 30 12:15:30.744020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.743544    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fab60c7793dfe7111457c92bac0feb6d6f3ed5ab064e0fb80dedb033ca688674"
Oct 30 12:15:30.745613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.745576    5060 generic.go:356] "Generic (PLEG): container finished" podID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerID="0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260" exitCode=0
Oct 30 12:15:30.745764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.745675    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" event={"ID":"85e29aab-e875-464e-b2e7-3cfb50e236a8","Type":"ContainerDied","Data":"0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"}
Oct 30 12:15:30.746308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.746268    5060 scope.go:122] "RemoveContainer" containerID="0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"
Oct 30 12:15:30.747952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.747924    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:30.748064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.747993    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:30.748064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:30.748032    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:31.766870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:31.766818    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" event={"ID":"85e29aab-e875-464e-b2e7-3cfb50e236a8","Type":"ContainerStarted","Data":"331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d"}
Oct 30 12:15:31.920212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:31.920175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:31.948845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:31.948786    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:15:31.949050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:31.949026    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 12:15:31.949112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:31.949069    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 12:15:32.949046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:32.949007    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 12:15:32.949517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:32.949063    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 12:15:33.047285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:33.047243    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:33.833923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:33.833886    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:33.949614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:33.949581    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 12:15:33.950034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:33.949634    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 12:15:34.268723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.268678    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:34.644317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.644245    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"]
Oct 30 12:15:34.644917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.644902    5060 cpu_manager.go:414] "containerMap: removing container" podUID="400d405b-a148-4d3d-a0a8-a7c294372c52" containerName="revoke-keys"
Oct 30 12:15:34.644969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.644919    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400d405b-a148-4d3d-a0a8-a7c294372c52" containerName="revoke-keys"
Oct 30 12:15:34.645039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.645030    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="400d405b-a148-4d3d-a0a8-a7c294372c52" containerName="revoke-keys"
Oct 30 12:15:34.650133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.650116    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.669386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.669360    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b15ec3-openshift-default-dockercfg-r9lmn\""
Oct 30 12:15:34.672003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.671976    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672291    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672334    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98q22\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672379    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672404    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672466    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672525    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.672794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.672632    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.702435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.702400    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"]
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774400    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774492    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774537    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774583    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98q22\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774634    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774661    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.830476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.774782    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.868480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.843945    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.932571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.871570    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.967647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.871783    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.981546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.872343    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.981912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.925826    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:34.982135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.926172    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.982288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.953890    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 12:15:34.992668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.992630    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 12:15:34.993276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.971951    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.993466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.980156    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98q22\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:34.993601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:34.956669    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:35.006476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.003237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token\") pod \"e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") " pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:35.125669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.125619    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:35.275147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.275115    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:35.635263 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:15:35.635232    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2f3122_ec0f_401f_b8b8_e37640160a8e.slice/crio-1d3be953223e7e80328093be5bc82874e5e3e7ead19291e053a42914b6585a82 WatchSource:0}: Error finding container 1d3be953223e7e80328093be5bc82874e5e3e7ead19291e053a42914b6585a82: Status 404 returned error can't find the container with id 1d3be953223e7e80328093be5bc82874e5e3e7ead19291e053a42914b6585a82
Oct 30 12:15:35.635946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.635888    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"]
Oct 30 12:15:35.638774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.638733    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:35.638887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.638822    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:35.638887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.638867    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:15:35.663987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.663897    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:35.949110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.949075    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 12:15:35.949338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:35.949123    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 12:15:36.013484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.012437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" event={"ID":"0c2f3122-ec0f-401f-b8b8-e37640160a8e","Type":"ContainerStarted","Data":"0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"}
Oct 30 12:15:36.013484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.012522    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" event={"ID":"0c2f3122-ec0f-401f-b8b8-e37640160a8e","Type":"ContainerStarted","Data":"1d3be953223e7e80328093be5bc82874e5e3e7ead19291e053a42914b6585a82"}
Oct 30 12:15:36.062715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.062659    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podStartSLOduration=2.06263844 podStartE2EDuration="2.06263844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:36.057882274 +0000 UTC m=+2739.852766767" watchObservedRunningTime="2026-10-30 12:15:36.06263844 +0000 UTC m=+2739.857522933"
Oct 30 12:15:36.275481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.275343    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:36.277582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.277516    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:15:36.277740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.277576    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:15:36.949413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.949374    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 12:15:36.949659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:36.949431    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 12:15:37.292530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:37.292106    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:37.379204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:37.379157    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:37.626735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:37.625366    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:37.771619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:37.771580    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:37.998477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:37.998366    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:15:38.002477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:37.999781    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:15:38.002477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:38.001054    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:15:38.075159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:38.075127    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:38.105302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:38.099500    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:15:38.239811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:38.239774    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:38.391239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:38.391145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:38.539569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:38.539536    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:39.379646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:39.379607    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:40.129053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:40.129006    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:40.276063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:40.276016    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:41.145692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:41.145662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:41.771306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:41.771277    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:15:42.894403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:42.894367    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:44.368707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:44.368658    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:44.512470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:44.512417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:44.661275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:44.661183    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:46.365566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:46.365521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:48.144343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:48.144299    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:49.037064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:49.037027    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:49.187776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:49.187681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:49.806953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:49.806921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:49.964920 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:15:49.964792    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:15:50.526595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:50.526553    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:50.676623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:50.676581    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:50.903935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:50.903831    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:51.088414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:51.088379    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:51.240709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:51.240673    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:51.495306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:51.495212    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:51.690169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:51.690131    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:52.153622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:52.153588    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:52.290296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:52.290265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:52.490173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:52.490140    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:53.561668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:53.561619    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:53.706018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:53.705987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:55.352427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:55.352400    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:57.046686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:57.046647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:15:58.719420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:15:58.719382    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:00.078421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:00.078385    5060 scope.go:122] "RemoveContainer" containerID="2e3f4461b811d055d66fc7eeb2d6fa20501e60684c7233199bae30720d8b192b"
Oct 30 12:16:00.090123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:00.090101    5060 scope.go:122] "RemoveContainer" containerID="d947386fe019235b3fd0bdb0db5c7580f559464a7150d72db1d499ba70f4c182"
Oct 30 12:16:00.458522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:00.458487    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:02.156911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:02.156863    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:03.813388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:03.813355    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:03.954556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:03.954518    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:04.101779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:04.101690    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:05.709434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:05.709382    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:07.450876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:07.450842    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:09.197639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:09.197605    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:10.868828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:10.868794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:12.564185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:12.564147    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:14.265718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:14.265681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:15.987784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:15.987735    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:16.125512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:16.125471    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:19.038163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:19.038126    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"]
Oct 30 12:16:19.050998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:19.050961    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e18ad2-27b9x"]
Oct 30 12:16:19.654332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:19.654297    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:19.800489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:19.800398    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:19.857232 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:16:19.857196    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:16:20.338706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:20.338673    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cbfb11-095e-4a6b-93c4-df330286262d" path="/var/lib/kubelet/pods/21cbfb11-095e-4a6b-93c4-df330286262d/volumes"
Oct 30 12:16:21.405883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:21.405845    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:21.956069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:21.956031    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:22.110835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:22.110800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:23.417393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:23.417343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:24.887490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:24.886784    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:25.120044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:25.120007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:26.961232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:26.961192    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:28.726549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:28.726507    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:29.611612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:29.611580    5060 generic.go:356] "Generic (PLEG): container finished" podID="1b033509-5c13-468f-8c29-28ff246b975f" containerID="1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89" exitCode=0
Oct 30 12:16:29.611846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:29.611637    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" event={"ID":"1b033509-5c13-468f-8c29-28ff246b975f","Type":"ContainerDied","Data":"1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"}
Oct 30 12:16:29.612069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:29.612054    5060 scope.go:122] "RemoveContainer" containerID="1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"
Oct 30 12:16:29.613977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:29.613945    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:16:29.614109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:29.614021    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:16:29.614109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:29.614054    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:16:30.035916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.035847    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"]
Oct 30 12:16:30.040598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.040564    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"]
Oct 30 12:16:30.044705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.044673    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5f53b7-z5wbh"]
Oct 30 12:16:30.049871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.049809    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c800233-chmbt"]
Oct 30 12:16:30.345090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.345003    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3e9406-2204-419c-9e15-35dd0c14b285" path="/var/lib/kubelet/pods/4b3e9406-2204-419c-9e15-35dd0c14b285/volumes"
Oct 30 12:16:30.345698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.345674    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94410c7d-6b0f-4ae2-8141-b86b2c2ce34b" path="/var/lib/kubelet/pods/94410c7d-6b0f-4ae2-8141-b86b2c2ce34b/volumes"
Oct 30 12:16:30.514990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.514954    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:30.623440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:30.623352    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" event={"ID":"1b033509-5c13-468f-8c29-28ff246b975f","Type":"ContainerStarted","Data":"575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244"}
Oct 30 12:16:31.275695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:31.275664    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:16:31.276114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:31.275821    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 12:16:31.276114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:31.275880    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 12:16:32.275851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:32.275817    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 12:16:32.276277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:32.275893    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 12:16:32.307831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:32.306653    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:33.276295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:33.276259    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 12:16:33.276815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:33.276330    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 12:16:34.065183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:34.065150    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:34.275720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:34.275687    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 12:16:34.275923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:34.275738    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 12:16:35.276091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:35.276058    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 12:16:35.276583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:35.276122    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 12:16:35.596260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:35.596168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:35.674524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:35.674491    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"]
Oct 30 12:16:35.791650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:35.791600    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:36.281064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:36.281035    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:16:36.281607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:36.281588    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:16:36.282328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:36.282308    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:16:37.579539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:37.574998    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:39.385247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:39.385210    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:40.675159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:40.675085    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh" podUID="84c28b2d-ffec-4485-810a-72337e356f05" containerName="payload-processing" containerID="cri-o://40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa" gracePeriod=25
Oct 30 12:16:40.960880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:40.960855    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:16:41.037500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.037445    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"]
Oct 30 12:16:41.042446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.042418    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06322cf-rxcgk"]
Oct 30 12:16:41.089729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.089697    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86\") pod \"84c28b2d-ffec-4485-810a-72337e356f05\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") "
Oct 30 12:16:41.089955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.089759    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials\") pod \"84c28b2d-ffec-4485-810a-72337e356f05\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") "
Oct 30 12:16:41.090026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.089965    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume\") pod \"84c28b2d-ffec-4485-810a-72337e356f05\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") "
Oct 30 12:16:41.090085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.090026    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume\") pod \"84c28b2d-ffec-4485-810a-72337e356f05\" (UID: \"84c28b2d-ffec-4485-810a-72337e356f05\") "
Oct 30 12:16:41.090343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.090314    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume" pod "84c28b2d-ffec-4485-810a-72337e356f05" (UID: "84c28b2d-ffec-4485-810a-72337e356f05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:16:41.090519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.090409    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume" pod "84c28b2d-ffec-4485-810a-72337e356f05" (UID: "84c28b2d-ffec-4485-810a-72337e356f05"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:16:41.090640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.090625    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:41.090696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.090643    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c28b2d-ffec-4485-810a-72337e356f05-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:41.092732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.092697    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials" pod "84c28b2d-ffec-4485-810a-72337e356f05" (UID: "84c28b2d-ffec-4485-810a-72337e356f05"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:16:41.092830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.092744    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86" pod "84c28b2d-ffec-4485-810a-72337e356f05" (UID: "84c28b2d-ffec-4485-810a-72337e356f05"). InnerVolumeSpecName "kube-api-access-j5t86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:16:41.115334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.115303    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:41.191891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.191811    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5t86\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-kube-api-access-j5t86\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:41.191891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.191838    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c28b2d-ffec-4485-810a-72337e356f05-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:41.719895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.719856    5060 generic.go:356] "Generic (PLEG): container finished" podID="84c28b2d-ffec-4485-810a-72337e356f05" containerID="40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa" exitCode=0
Oct 30 12:16:41.720406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.719900    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"
Oct 30 12:16:41.720406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.719938    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh" event={"ID":"84c28b2d-ffec-4485-810a-72337e356f05","Type":"ContainerDied","Data":"40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa"}
Oct 30 12:16:41.720406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.720006    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh" event={"ID":"84c28b2d-ffec-4485-810a-72337e356f05","Type":"ContainerDied","Data":"c8386db46c64ab4b893562d5dec030fd9db4d5fb71630f0e13af6247c22fcf28"}
Oct 30 12:16:41.720406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.720037    5060 scope.go:122] "RemoveContainer" containerID="40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa"
Oct 30 12:16:41.732541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.732521    5060 scope.go:122] "RemoveContainer" containerID="40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa"
Oct 30 12:16:41.732867 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:16:41.732850    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa\": container with ID starting with 40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa not found: ID does not exist" containerID="40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa"
Oct 30 12:16:41.732920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.732876    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa"} err="failed to get container status \"40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa\": rpc error: code = NotFound desc = could not find container \"40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa\": container with ID starting with 40ddc5fbeb6093fffea41392a791def2b42809edf62b3e2e60b901eda3e504fa not found: ID does not exist"
Oct 30 12:16:41.773472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.773429    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"]
Oct 30 12:16:41.786026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:41.785998    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d77zbnh"]
Oct 30 12:16:42.335034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:42.335001    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c28b2d-ffec-4485-810a-72337e356f05" path="/var/lib/kubelet/pods/84c28b2d-ffec-4485-810a-72337e356f05/volumes"
Oct 30 12:16:42.335479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:42.335443    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beec1a4e-7269-4d70-a5ba-abf40f031a0b" path="/var/lib/kubelet/pods/beec1a4e-7269-4d70-a5ba-abf40f031a0b/volumes"
Oct 30 12:16:42.865062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:42.865026    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:44.665018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:44.664980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:46.526585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:46.526542    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:47.887673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:47.887633    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:48.355145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:48.355112    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:49.858664 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:16:49.858625    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:16:50.058637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:50.058603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:52.065729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:52.065682    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:52.619754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:52.619722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:53.936351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:53.936304    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:54.804478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.803467    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"]
Oct 30 12:16:54.804760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.804729    5060 cpu_manager.go:414] "containerMap: removing container" podUID="84c28b2d-ffec-4485-810a-72337e356f05" containerName="payload-processing"
Oct 30 12:16:54.804852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.804762    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c28b2d-ffec-4485-810a-72337e356f05" containerName="payload-processing"
Oct 30 12:16:54.806398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.804928    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c28b2d-ffec-4485-810a-72337e356f05" containerName="payload-processing"
Oct 30 12:16:54.819547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.819516    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"]
Oct 30 12:16:54.819867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.819847    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:54.847949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.847925    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:16:54.848158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.847926    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:16:54.848282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.848070    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:16:54.848381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.848113    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:16:54.954404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.954363    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:54.954849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.954419    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2m7\" (UniqueName: \"kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:54.954849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:54.954595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.034379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.034344    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:55.055894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.055797    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2m7\" (UniqueName: \"kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.056081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.055933    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.056147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.056105    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.056761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.056735    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.058767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.058742    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.101479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.099551    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2m7\" (UniqueName: \"kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7\") pod \"maas-api-revoke-sub-7f1a53f3-nfx7d\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.140561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.140530    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:55.205102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.205027    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:55.860846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.860700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:55.961294 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:16:55.961266    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10aa4d9_934a_4c7c_8e54_bec3a6b6407c.slice/crio-9bc3f33e3e884b9efd7f6e25031f3f796f1e77e40c5e957d769fc7678e96e316 WatchSource:0}: Error finding container 9bc3f33e3e884b9efd7f6e25031f3f796f1e77e40c5e957d769fc7678e96e316: Status 404 returned error can't find the container with id 9bc3f33e3e884b9efd7f6e25031f3f796f1e77e40c5e957d769fc7678e96e316
Oct 30 12:16:55.965372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:55.965334    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"]
Oct 30 12:16:56.915761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:56.915664    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d" event={"ID":"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c","Type":"ContainerStarted","Data":"3af03781c107ee84b6231b6375ab7f504e760a0174e4a0291175310320e9c6ea"}
Oct 30 12:16:56.916029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:56.915998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d" event={"ID":"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c","Type":"ContainerStarted","Data":"9bc3f33e3e884b9efd7f6e25031f3f796f1e77e40c5e957d769fc7678e96e316"}
Oct 30 12:16:56.957125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:56.957074    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d" podStartSLOduration=2.957060035 podStartE2EDuration="2.957060035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:56.956384086 +0000 UTC m=+2820.751268585" watchObservedRunningTime="2026-10-30 12:16:56.957060035 +0000 UTC m=+2820.751944576"
Oct 30 12:16:57.208411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.208372    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:57.434209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.434160    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"]
Oct 30 12:16:57.440813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.440779    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.451920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.451887    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"]
Oct 30 12:16:57.456965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.456935    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"kube-root-ca.crt\""
Oct 30 12:16:57.457183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.457163    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"payload-processing-external-model-plugins-e2e-extmod-a-737f64\""
Oct 30 12:16:57.457285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.457206    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"openshift-service-ca.crt\""
Oct 30 12:16:57.457285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.457255    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"routing-overlay\""
Oct 30 12:16:57.549535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.546735    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"agc-cred-e941953a13cc\""
Oct 30 12:16:57.549535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.546879    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"payload-processing-external-model-e2e--f69e4050-dockercfg-9wvg4\""
Oct 30 12:16:57.601542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.601496    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.601918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.601891    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.602223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.602200    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.606536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.606498    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865pt\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.711479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.711404    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.711710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.711505    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.711710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.711610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.711710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.711694    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865pt\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.729537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.713031    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.729537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.713634    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.729537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.720257    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.741537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.741507    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-865pt\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.759901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.759873    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:57.780480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.776810    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:57.951559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.951529    5060 generic.go:356] "Generic (PLEG): container finished" podID="b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" containerID="3af03781c107ee84b6231b6375ab7f504e760a0174e4a0291175310320e9c6ea" exitCode=0
Oct 30 12:16:57.951800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:57.951784    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d" event={"ID":"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c","Type":"ContainerDied","Data":"3af03781c107ee84b6231b6375ab7f504e760a0174e4a0291175310320e9c6ea"}
Oct 30 12:16:58.200511 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:16:58.200471    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0bb194_879d_4555_b8f8_c3e22a3f3b0f.slice/crio-33418504ae0e74aec682e876687ca90717e4bc2d7e807dd72e22f3f893776055 WatchSource:0}: Error finding container 33418504ae0e74aec682e876687ca90717e4bc2d7e807dd72e22f3f893776055: Status 404 returned error can't find the container with id 33418504ae0e74aec682e876687ca90717e4bc2d7e807dd72e22f3f893776055
Oct 30 12:16:58.201289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:58.201263    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"]
Oct 30 12:16:58.964156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:58.964120    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb" event={"ID":"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f","Type":"ContainerStarted","Data":"e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c"}
Oct 30 12:16:58.964734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:58.964180    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb" event={"ID":"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f","Type":"ContainerStarted","Data":"33418504ae0e74aec682e876687ca90717e4bc2d7e807dd72e22f3f893776055"}
Oct 30 12:16:58.964734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:58.964206    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:16:59.009099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.009028    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb" podStartSLOduration=2.009009101 podStartE2EDuration="2.009009101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:59.008203836 +0000 UTC m=+2822.803088331" watchObservedRunningTime="2026-10-30 12:16:59.009009101 +0000 UTC m=+2822.803893595"
Oct 30 12:16:59.163165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.163136    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:59.351531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.350985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7\" (UniqueName: \"kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7\") pod \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") "
Oct 30 12:16:59.351722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.351563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca\") pod \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") "
Oct 30 12:16:59.355476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.351872    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca\") pod \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\" (UID: \"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c\") "
Oct 30 12:16:59.357505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.356669    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca" pod "b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" (UID: "b10aa4d9-934a-4c7c-8e54-bec3a6b6407c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:16:59.357505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.356971    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:59.362396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.362163    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:59.366355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.366302    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7" pod "b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" (UID: "b10aa4d9-934a-4c7c-8e54-bec3a6b6407c"). InnerVolumeSpecName "kube-api-access-cc2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:16:59.412219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.411618    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca" pod "b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" (UID: "b10aa4d9-934a-4c7c-8e54-bec3a6b6407c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:16:59.463736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.463700    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2m7\" (UniqueName: \"kubernetes.io/projected/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-kube-api-access-cc2m7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:59.463736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.463731    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:16:59.573120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.573087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:59.719474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.718481    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"]
Oct 30 12:16:59.719678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.719650    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" containerName="revoke-keys"
Oct 30 12:16:59.719678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.719669    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" containerName="revoke-keys"
Oct 30 12:16:59.720007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.719805    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" containerName="revoke-keys"
Oct 30 12:16:59.732505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.729721    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.750364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.750333    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bae155-openshift-default-dockercfg-lv9sp\""
Oct 30 12:16:59.750681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.750650    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"]
Oct 30 12:16:59.767851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.767818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.767898    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.767985    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.768086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrrf\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.768202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.768241    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.768300    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.768356    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.768428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.768413    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.806884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.806850    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:59.870378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870320    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.870622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870433    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.870622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870574    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.870772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870675    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrrf\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.870831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870799    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.870897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870833    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.870961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870897    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.871024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.870957    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.871024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.871011    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.872126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.871129    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.872126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.871557    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.872126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.871849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.872126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.871851    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.872126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.872025    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.881834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.881764    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.905914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.897160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.911010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.910973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrrf\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.919129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.919044    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo\") pod \"e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") " pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:16:59.962502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.958792    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:16:59.992526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.990540    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"
Oct 30 12:16:59.997235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.993942    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d" event={"ID":"b10aa4d9-934a-4c7c-8e54-bec3a6b6407c","Type":"ContainerDied","Data":"9bc3f33e3e884b9efd7f6e25031f3f796f1e77e40c5e957d769fc7678e96e316"}
Oct 30 12:16:59.997235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:16:59.994006    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc3f33e3e884b9efd7f6e25031f3f796f1e77e40c5e957d769fc7678e96e316"
Oct 30 12:17:00.068873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:00.068254    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:17:00.132475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:00.131416    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:00.453163 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:17:00.453128    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abe7791_6018_475c_b11e_89f03f1206a3.slice/crio-317d3bae4ff1f821a6083a22bc8e259af23bdf0fabecc85fe1069f92d90964a3 WatchSource:0}: Error finding container 317d3bae4ff1f821a6083a22bc8e259af23bdf0fabecc85fe1069f92d90964a3: Status 404 returned error can't find the container with id 317d3bae4ff1f821a6083a22bc8e259af23bdf0fabecc85fe1069f92d90964a3
Oct 30 12:17:00.456771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:00.456734    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:17:00.456887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:00.456821    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:17:00.456887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:00.456853    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"]
Oct 30 12:17:00.456887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:00.456868    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:17:01.005749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.005710    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" event={"ID":"8abe7791-6018-475c-b11e-89f03f1206a3","Type":"ContainerStarted","Data":"ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"}
Oct 30 12:17:01.006377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.006351    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" event={"ID":"8abe7791-6018-475c-b11e-89f03f1206a3","Type":"ContainerStarted","Data":"317d3bae4ff1f821a6083a22bc8e259af23bdf0fabecc85fe1069f92d90964a3"}
Oct 30 12:17:01.053183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.053119    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podStartSLOduration=2.053100974 podStartE2EDuration="2.053100974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:01.051583577 +0000 UTC m=+2824.846468071" watchObservedRunningTime="2026-10-30 12:17:01.053100974 +0000 UTC m=+2824.847985470"
Oct 30 12:17:01.069244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.069210    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:17:01.075283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.075252    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:17:01.285227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.285145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:01.431941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.431900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:01.678727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.678622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:01.855698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:01.855651    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:02.013678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:02.013642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:02.019520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:02.019485    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:17:02.020508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:02.020442    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:17:02.173672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:02.173632    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:02.766284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:02.766253    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:02.914548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:02.914385    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:03.105529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.097528    5060 generic.go:356] "Generic (PLEG): container finished" podID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerID="0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53" exitCode=0
Oct 30 12:17:03.105529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.098559    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" event={"ID":"0c2f3122-ec0f-401f-b8b8-e37640160a8e","Type":"ContainerDied","Data":"0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"}
Oct 30 12:17:03.105529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.098988    5060 scope.go:122] "RemoveContainer" containerID="0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"
Oct 30 12:17:03.107313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.107269    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:17:03.107472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.107371    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:17:03.107472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.107420    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:17:03.706819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.706766    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:03.863522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:03.863039    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:04.132631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:04.131788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" event={"ID":"0c2f3122-ec0f-401f-b8b8-e37640160a8e","Type":"ContainerStarted","Data":"e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762"}
Oct 30 12:17:04.210183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:04.210140    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:04.281474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:04.280667    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:17:04.295477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:04.286579    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:17:04.295477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:04.286655    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:17:05.275761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:05.275725    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:17:05.276281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:05.275781    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:17:05.821650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:05.821560    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:06.081535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:06.059430    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:06.330645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:06.325134    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:17:06.330645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:06.325205    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:17:06.393579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:06.368226    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:06.621591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:06.621320    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:06.912049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:06.912011    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:07.055309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.055270    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:07.245024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.244990    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:07.283668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.283631    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:17:07.283952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.283923    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:17:07.403369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.403333    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:07.540890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.540797    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:07.691488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.686988    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:07.837790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:07.837705    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:08.075733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:08.075700    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:17:08.275645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:08.275613    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:17:08.275838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:08.275676    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:17:08.354307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:08.354272    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:08.508480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:08.507709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:09.275575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.275543    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 12:17:09.275758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.275602    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 12:17:09.402423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.402385    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:09.626361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.626279    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"]
Oct 30 12:17:09.630922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.630903    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.651977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.651946    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"]
Oct 30 12:17:09.663049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.663017    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"openshift-service-ca.crt\""
Oct 30 12:17:09.663243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.663120    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"routing-overlay\""
Oct 30 12:17:09.663243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.663127    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"agc-cred-0c9e9832e52d\""
Oct 30 12:17:09.663243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.663183    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-e2e--8475db1b-dockercfg-qmscv\""
Oct 30 12:17:09.663243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.663191    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f8f132\""
Oct 30 12:17:09.663444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.663267    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"kube-root-ca.crt\""
Oct 30 12:17:09.708614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.708557    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:09.712887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.712852    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.713120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.713097    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2qd\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.713259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.713243    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.713398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.713376    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.841272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.841231    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.841695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.841668    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.841978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.841959    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.842133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.842116    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2qd\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.855555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.845301    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.855555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.845872    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.877532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.875864    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.954131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.954094    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2qd\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:09.999726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:09.999686    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:17:10.246414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:10.246380    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:10.549967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:10.549873    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:17:10.551523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:10.551445    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:17:10.566790 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:17:10.566761    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1549e043_1259_4218_b43f_d27168ec7574.slice/crio-86a5ff36c3279bc76571619ce03c083ecffcf9b68bb13f2b0e9abc327d3519b4 WatchSource:0}: Error finding container 86a5ff36c3279bc76571619ce03c083ecffcf9b68bb13f2b0e9abc327d3519b4: Status 404 returned error can't find the container with id 86a5ff36c3279bc76571619ce03c083ecffcf9b68bb13f2b0e9abc327d3519b4
Oct 30 12:17:10.568943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:10.568923    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"]
Oct 30 12:17:11.036474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.032982    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:11.204111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.204075    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:11.399202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.399094    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" event={"ID":"1549e043-1259-4218-b43f-d27168ec7574","Type":"ContainerStarted","Data":"d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7"}
Oct 30 12:17:11.399202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.399168    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:11.399202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.399194    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" event={"ID":"1549e043-1259-4218-b43f-d27168ec7574","Type":"ContainerStarted","Data":"86a5ff36c3279bc76571619ce03c083ecffcf9b68bb13f2b0e9abc327d3519b4"}
Oct 30 12:17:11.416033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.416002    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:11.441386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.441319    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" podStartSLOduration=2.441302016 podStartE2EDuration="2.441302016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:11.440120848 +0000 UTC m=+2835.235005344" watchObservedRunningTime="2026-10-30 12:17:11.441302016 +0000 UTC m=+2835.236186514"
Oct 30 12:17:11.507475 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:11.480862    5060 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:11.507475 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:11.480887    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:11.507475 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:11.480960    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials podName:1549e043-1259-4218-b43f-d27168ec7574 nodeName:}" failed. No retries permitted until 2026-10-30 12:17:11.980940666 +0000 UTC m=+2835.775825151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" (UID: "1549e043-1259-4218-b43f-d27168ec7574") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:11.577475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.576255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:11.720324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:11.718589    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:12.007406 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:12.007323    5060 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:12.007406 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:12.007353    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:12.007636 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:12.007422    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials podName:1549e043-1259-4218-b43f-d27168ec7574 nodeName:}" failed. No retries permitted until 2026-10-30 12:17:13.007406353 +0000 UTC m=+2836.802290830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" (UID: "1549e043-1259-4218-b43f-d27168ec7574") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:13.021440 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:13.021405    5060 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:13.021440 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:13.021433    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:13.022000 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:13.021516    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials podName:1549e043-1259-4218-b43f-d27168ec7574 nodeName:}" failed. No retries permitted until 2026-10-30 12:17:15.021498688 +0000 UTC m=+2838.816383180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" (UID: "1549e043-1259-4218-b43f-d27168ec7574") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:13.448365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:13.448331    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:15.050961 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:15.050913    5060 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:15.050961 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:15.050949    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:15.051400 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:15.051018    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials podName:1549e043-1259-4218-b43f-d27168ec7574 nodeName:}" failed. No retries permitted until 2026-10-30 12:17:19.050999121 +0000 UTC m=+2842.845883612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" (UID: "1549e043-1259-4218-b43f-d27168ec7574") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:15.124384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:15.124339    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:16.794356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:16.794319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:18.560719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:18.560677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:19.110900 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:19.110858    5060 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:19.110900 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:19.110886    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:19.111186 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:19.110948    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials podName:1549e043-1259-4218-b43f-d27168ec7574 nodeName:}" failed. No retries permitted until 2026-10-30 12:17:27.110929337 +0000 UTC m=+2850.905813809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" (UID: "1549e043-1259-4218-b43f-d27168ec7574") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:19.857570 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:19.857528    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:17:20.311532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:20.311494    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:22.064399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:22.061849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:22.411488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:22.411374    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:23.870928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:23.870890    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:24.017184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:24.017135    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"]
Oct 30 12:17:25.715478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:25.714811    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:27.193342 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:27.193301    5060 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:27.193342 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:27.193327    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:27.193976 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:27.193389    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials podName:1549e043-1259-4218-b43f-d27168ec7574 nodeName:}" failed. No retries permitted until 2026-10-30 12:17:43.193372627 +0000 UTC m=+2866.988257111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" (UID: "1549e043-1259-4218-b43f-d27168ec7574") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:17:27.561369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:27.561332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:29.017794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.017742    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" podUID="1549e043-1259-4218-b43f-d27168ec7574" containerName="payload-processing" containerID="cri-o://d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7" gracePeriod=25
Oct 30 12:17:29.333502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.333470    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:29.343958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.343921    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:29.425509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.425445    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume\") pod \"1549e043-1259-4218-b43f-d27168ec7574\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") "
Oct 30 12:17:29.425702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.425651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume\") pod \"1549e043-1259-4218-b43f-d27168ec7574\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") "
Oct 30 12:17:29.425702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.425683    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials\") pod \"1549e043-1259-4218-b43f-d27168ec7574\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") "
Oct 30 12:17:29.425851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.425770    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd\") pod \"1549e043-1259-4218-b43f-d27168ec7574\" (UID: \"1549e043-1259-4218-b43f-d27168ec7574\") "
Oct 30 12:17:29.427475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.427417    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume" pod "1549e043-1259-4218-b43f-d27168ec7574" (UID: "1549e043-1259-4218-b43f-d27168ec7574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:29.427602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.427549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume" pod "1549e043-1259-4218-b43f-d27168ec7574" (UID: "1549e043-1259-4218-b43f-d27168ec7574"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:29.429944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.429911    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials" pod "1549e043-1259-4218-b43f-d27168ec7574" (UID: "1549e043-1259-4218-b43f-d27168ec7574"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:29.430068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.429984    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd" pod "1549e043-1259-4218-b43f-d27168ec7574" (UID: "1549e043-1259-4218-b43f-d27168ec7574"). InnerVolumeSpecName "kube-api-access-mv2qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:29.527215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.527183    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:29.527215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.527211    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:29.527215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.527222    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2qd\" (UniqueName: \"kubernetes.io/projected/1549e043-1259-4218-b43f-d27168ec7574-kube-api-access-mv2qd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:29.527500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.527230    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1549e043-1259-4218-b43f-d27168ec7574-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:29.585631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.585538    5060 generic.go:356] "Generic (PLEG): container finished" podID="1549e043-1259-4218-b43f-d27168ec7574" containerID="d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7" exitCode=0
Oct 30 12:17:29.585631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.585599    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"
Oct 30 12:17:29.585826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.585631    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" event={"ID":"1549e043-1259-4218-b43f-d27168ec7574","Type":"ContainerDied","Data":"d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7"}
Oct 30 12:17:29.585826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.585704    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj" event={"ID":"1549e043-1259-4218-b43f-d27168ec7574","Type":"ContainerDied","Data":"86a5ff36c3279bc76571619ce03c083ecffcf9b68bb13f2b0e9abc327d3519b4"}
Oct 30 12:17:29.585826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.585733    5060 scope.go:122] "RemoveContainer" containerID="d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7"
Oct 30 12:17:29.602673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.602642    5060 scope.go:122] "RemoveContainer" containerID="d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7"
Oct 30 12:17:29.603029 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:29.603000    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7\": container with ID starting with d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7 not found: ID does not exist" containerID="d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7"
Oct 30 12:17:29.603078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.603042    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7"} err="failed to get container status \"d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7\": rpc error: code = NotFound desc = could not find container \"d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7\": container with ID starting with d6e0cee6d934d61543502ac026d0af811a6aac3958ecc98151ca46e390393eb7 not found: ID does not exist"
Oct 30 12:17:29.648233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.648194    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"]
Oct 30 12:17:29.653581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:29.653543    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885mtbmj"]
Oct 30 12:17:30.335844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:30.335812    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1549e043-1259-4218-b43f-d27168ec7574" path="/var/lib/kubelet/pods/1549e043-1259-4218-b43f-d27168ec7574/volumes"
Oct 30 12:17:31.091276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:31.091233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:33.183836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:33.183797    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:33.197258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:33.197227    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"]
Oct 30 12:17:33.326342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:33.326306    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:35.274186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:35.274145    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:37.079835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:37.079798    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:38.197701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.197656    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb" podUID="bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" containerName="payload-processing" containerID="cri-o://e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c" gracePeriod=25
Oct 30 12:17:38.492392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.492360    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:17:38.552803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.552772    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume\") pod \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") "
Oct 30 12:17:38.552983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.552912    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt\") pod \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") "
Oct 30 12:17:38.552983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.552951    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume\") pod \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") "
Oct 30 12:17:38.553122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.553062    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials\") pod \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\" (UID: \"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f\") "
Oct 30 12:17:38.553431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.553219    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume" pod "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" (UID: "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:38.553431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.553402    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume" pod "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" (UID: "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:38.553750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.553724    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:38.553852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.553760    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:38.555514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.555490    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt" pod "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" (UID: "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f"). InnerVolumeSpecName "kube-api-access-865pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:38.555852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.555831    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials" pod "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" (UID: "bd0bb194-879d-4555-b8f8-c3e22a3f3b0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:38.655278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.655238    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-865pt\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-kube-api-access-865pt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:38.655278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.655276    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:38.689169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.689132    5060 generic.go:356] "Generic (PLEG): container finished" podID="bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" containerID="e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c" exitCode=0
Oct 30 12:17:38.689358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.689180    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"
Oct 30 12:17:38.689358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.689210    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb" event={"ID":"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f","Type":"ContainerDied","Data":"e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c"}
Oct 30 12:17:38.689358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.689259    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb" event={"ID":"bd0bb194-879d-4555-b8f8-c3e22a3f3b0f","Type":"ContainerDied","Data":"33418504ae0e74aec682e876687ca90717e4bc2d7e807dd72e22f3f893776055"}
Oct 30 12:17:38.689358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.689285    5060 scope.go:122] "RemoveContainer" containerID="e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c"
Oct 30 12:17:38.700786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.700768    5060 scope.go:122] "RemoveContainer" containerID="e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c"
Oct 30 12:17:38.701051 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:38.701027    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c\": container with ID starting with e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c not found: ID does not exist" containerID="e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c"
Oct 30 12:17:38.701125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.701061    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c"} err="failed to get container status \"e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c\": rpc error: code = NotFound desc = could not find container \"e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c\": container with ID starting with e99c4781557376ba894b4dc632947f3d089a59696930fdc7b005282d0b471f7c not found: ID does not exist"
Oct 30 12:17:38.743541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.743464    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"]
Oct 30 12:17:38.748800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.748777    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-5f47gw5sb"]
Oct 30 12:17:38.957301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:38.957267    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:40.335021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:40.334990    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" path="/var/lib/kubelet/pods/bd0bb194-879d-4555-b8f8-c3e22a3f3b0f/volumes"
Oct 30 12:17:40.771722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:40.771657    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:41.474087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:41.474051    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:41.631425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:41.631386    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:42.654752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.654712    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:42.822429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.822389    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"]
Oct 30 12:17:42.823182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.823159    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" containerName="payload-processing"
Oct 30 12:17:42.823302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.823182    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" containerName="payload-processing"
Oct 30 12:17:42.823302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.823228    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1549e043-1259-4218-b43f-d27168ec7574" containerName="payload-processing"
Oct 30 12:17:42.823302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.823236    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1549e043-1259-4218-b43f-d27168ec7574" containerName="payload-processing"
Oct 30 12:17:42.823516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.823397    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1549e043-1259-4218-b43f-d27168ec7574" containerName="payload-processing"
Oct 30 12:17:42.823516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.823422    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0bb194-879d-4555-b8f8-c3e22a3f3b0f" containerName="payload-processing"
Oct 30 12:17:42.828340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.828318    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:42.843620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.843593    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"]
Oct 30 12:17:42.843760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.843670    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"agc-cred-010fd4c5b4d7\""
Oct 30 12:17:42.850633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.850609    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"payload-processing-external-model-e2e--9f9529d9-dockercfg-fdxmq\""
Oct 30 12:17:42.850815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.850774    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"agc-cred-097f9a689008\""
Oct 30 12:17:42.850915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.850899    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4f7443\""
Oct 30 12:17:42.851049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.851033    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"openshift-service-ca.crt\""
Oct 30 12:17:42.856107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.856085    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"kube-root-ca.crt\""
Oct 30 12:17:42.856258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.856239    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"routing-overlay\""
Oct 30 12:17:42.917061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.916951    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:42.917061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.917036    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:42.917270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.917157    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpg5s\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:42.917339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:42.917318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.018988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.018952    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.019170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.019033    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.019170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.019064    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpg5s\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.019170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.019145    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.019807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.019776    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.019937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.019915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.022229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.022204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.042830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.042803    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpg5s\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.145918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.145888    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.451078 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:17:43.451046    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d6c746_aff5_41b1_aeb6_49bfe0aedb9b.slice/crio-fd446a52b5060be68af6a73c58089c69445a3ecb16056720b7057ed1ea1b8d55 WatchSource:0}: Error finding container fd446a52b5060be68af6a73c58089c69445a3ecb16056720b7057ed1ea1b8d55: Status 404 returned error can't find the container with id fd446a52b5060be68af6a73c58089c69445a3ecb16056720b7057ed1ea1b8d55
Oct 30 12:17:43.454831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.454806    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"]
Oct 30 12:17:43.742563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.742445    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh" event={"ID":"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b","Type":"ContainerStarted","Data":"57f01c91446ee5e5548428089bb00253410fbabecdec21cee3b6a483eef78e71"}
Oct 30 12:17:43.742563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.742542    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh" event={"ID":"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b","Type":"ContainerStarted","Data":"fd446a52b5060be68af6a73c58089c69445a3ecb16056720b7057ed1ea1b8d55"}
Oct 30 12:17:43.743005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.742574    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:43.788821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:43.788754    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh" podStartSLOduration=1.788736238 podStartE2EDuration="1.788736238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:43.784809312 +0000 UTC m=+2867.579693808" watchObservedRunningTime="2026-10-30 12:17:43.788736238 +0000 UTC m=+2867.583620733"
Oct 30 12:17:44.373935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:44.373733    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:46.126020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:46.125980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:46.275029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:46.274436    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:47.970478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:47.967994    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:49.686009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:49.685964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:49.858329 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:17:49.858291    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:17:51.461612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:51.461572    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:52.374413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:52.374360    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:52.520312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:52.520276    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:52.823472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:52.822975    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:53.339474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.330492    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:53.829169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.829104    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"]
Oct 30 12:17:53.832183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.832158    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:53.862721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.862689    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"]
Oct 30 12:17:53.867297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.867277    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.882574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.882551    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"agc-cred-3e3ad2495bdc\""
Oct 30 12:17:53.886784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.886753    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"]
Oct 30 12:17:53.892129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.892100    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.892386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.892363    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.892533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.892434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pc2x\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.892737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.892715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.993834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.993800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.994059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.993852    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.994059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.993923    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pc2x\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.994179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.994158    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.994808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.994776    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.994918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.994880    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:53.996717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:53.996692    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:54.017916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.017886    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pc2x\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:54.185184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.185083    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:54.496576 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:17:54.496538    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf173f1ce_9016_43f6_8618_f1a2f94f74e9.slice/crio-fc0e26346b56ee7d7ebebac960b17148d2d1cd25022963452d51f1cb6804753e WatchSource:0}: Error finding container fc0e26346b56ee7d7ebebac960b17148d2d1cd25022963452d51f1cb6804753e: Status 404 returned error can't find the container with id fc0e26346b56ee7d7ebebac960b17148d2d1cd25022963452d51f1cb6804753e
Oct 30 12:17:54.496778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.496757    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"]
Oct 30 12:17:54.909991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.909888    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4" event={"ID":"f173f1ce-9016-43f6-8618-f1a2f94f74e9","Type":"ContainerStarted","Data":"586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70"}
Oct 30 12:17:54.909991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.909950    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4" event={"ID":"f173f1ce-9016-43f6-8618-f1a2f94f74e9","Type":"ContainerStarted","Data":"fc0e26346b56ee7d7ebebac960b17148d2d1cd25022963452d51f1cb6804753e"}
Oct 30 12:17:54.909991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.909980    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:17:54.954582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:54.954532    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4" podStartSLOduration=1.9545167079999999 podStartE2EDuration="1.954516708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:54.953216392 +0000 UTC m=+2878.748100886" watchObservedRunningTime="2026-10-30 12:17:54.954516708 +0000 UTC m=+2878.749401201"
Oct 30 12:17:55.265137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:55.265104    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:57.085231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:57.085196    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:58.830287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:58.830228    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh" podUID="68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" containerName="payload-processing" containerID="cri-o://57f01c91446ee5e5548428089bb00253410fbabecdec21cee3b6a483eef78e71" gracePeriod=25
Oct 30 12:17:58.890657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:58.890614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:17:58.983322 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:58.983287    5060 generic.go:356] "Generic (PLEG): container finished" podID="68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" containerID="57f01c91446ee5e5548428089bb00253410fbabecdec21cee3b6a483eef78e71" exitCode=0
Oct 30 12:17:58.983531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:58.983353    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh" event={"ID":"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b","Type":"ContainerDied","Data":"57f01c91446ee5e5548428089bb00253410fbabecdec21cee3b6a483eef78e71"}
Oct 30 12:17:59.118605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.118575    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:59.184332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.184298    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s\") pod \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") "
Oct 30 12:17:59.184621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.184411    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume\") pod \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") "
Oct 30 12:17:59.184865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.184831    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume" pod "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" (UID: "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:59.185146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.185127    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume\") pod \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") "
Oct 30 12:17:59.185244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.185199    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials\") pod \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\" (UID: \"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b\") "
Oct 30 12:17:59.185712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.185679    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume" pod "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" (UID: "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:59.186445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.186422    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:59.186445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.186468    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:59.187529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.187504    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s" pod "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" (UID: "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b"). InnerVolumeSpecName "kube-api-access-qpg5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:59.187785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.187763    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials" pod "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" (UID: "68d6c746-aff5-41b1-aeb6-49bfe0aedb9b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:59.288559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.288521    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:59.288559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.288557    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpg5s\" (UniqueName: \"kubernetes.io/projected/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b-kube-api-access-qpg5s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:17:59.995912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.995876    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"
Oct 30 12:17:59.996333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.995879    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh" event={"ID":"68d6c746-aff5-41b1-aeb6-49bfe0aedb9b","Type":"ContainerDied","Data":"fd446a52b5060be68af6a73c58089c69445a3ecb16056720b7057ed1ea1b8d55"}
Oct 30 12:17:59.996333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:17:59.996059    5060 scope.go:122] "RemoveContainer" containerID="57f01c91446ee5e5548428089bb00253410fbabecdec21cee3b6a483eef78e71"
Oct 30 12:18:00.050484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.050435    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"]
Oct 30 12:18:00.062858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.062831    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68nbhh"]
Oct 30 12:18:00.261611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.261532    5060 scope.go:122] "RemoveContainer" containerID="b7b25baf7040e7d147c13ce9052d78b4768f65be4a0bbb7d3426b0e946adf01d"
Oct 30 12:18:00.272429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.272408    5060 scope.go:122] "RemoveContainer" containerID="3fa3b23389fd56a782e013b84c7a249d1f33317a3c0cfa0e46c56effc919d327"
Oct 30 12:18:00.284954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.284932    5060 scope.go:122] "RemoveContainer" containerID="0db2e5e203d563e01c6bd3ba3add0cb73e011adbdb8aeb82e007ea8baba8f199"
Oct 30 12:18:00.297772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.297750    5060 scope.go:122] "RemoveContainer" containerID="930eaad18eada84d389a634f776f17b9e514569fddb1cd9f8edc152e8c47208e"
Oct 30 12:18:00.335764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.335736    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" path="/var/lib/kubelet/pods/68d6c746-aff5-41b1-aeb6-49bfe0aedb9b/volumes"
Oct 30 12:18:00.590844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:00.590741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:02.272049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:02.272015    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:04.029308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:04.029271    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:05.827221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:05.827184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:05.923648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:05.923618    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:18:07.660419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:07.660381    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:09.446262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:09.446216    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:11.110153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:11.110111    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:12.875920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:12.875882    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:14.895827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:14.895788    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:16.582279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:16.582230    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:17.032234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:17.032189    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"]
Oct 30 12:18:17.044725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:17.044695    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9036b50a-kt64x"]
Oct 30 12:18:18.319292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:18.319255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:18.340648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:18.340617    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67024988-8e6f-48a3-b4a8-ba3896f18927" path="/var/lib/kubelet/pods/67024988-8e6f-48a3-b4a8-ba3896f18927/volumes"
Oct 30 12:18:19.857179 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:18:19.857139    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:18:20.064687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:20.064642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:21.849208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:21.849173    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:23.543789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:23.543749    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:25.228024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:25.227980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:26.033567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:26.033522    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"]
Oct 30 12:18:26.045836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:26.045806    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6950e800-f2zb6"]
Oct 30 12:18:26.339963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:26.339885    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30091c2-e6e3-4a8f-a070-c1b4721fcc8e" path="/var/lib/kubelet/pods/d30091c2-e6e3-4a8f-a070-c1b4721fcc8e/volumes"
Oct 30 12:18:26.963097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:26.963045    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:28.839151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:28.839066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:29.804638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:29.804601    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:29.958475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:29.952269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:30.670853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.670809    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:30.924836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.924752    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"]
Oct 30 12:18:30.926048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.926022    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" containerName="payload-processing"
Oct 30 12:18:30.926218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.926205    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" containerName="payload-processing"
Oct 30 12:18:30.926565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.926547    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d6c746-aff5-41b1-aeb6-49bfe0aedb9b" containerName="payload-processing"
Oct 30 12:18:30.931854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.931830    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"]
Oct 30 12:18:30.932103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.932088    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:30.962536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.962508    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"openshift-service-ca.crt\""
Oct 30 12:18:30.962963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.962529    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"kube-root-ca.crt\""
Oct 30 12:18:30.962963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.962647    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-59b8a9\""
Oct 30 12:18:30.962963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.962676    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"payload-processing-external-model-e2e--fe4784b3-dockercfg-dnr24\""
Oct 30 12:18:30.962963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.962735    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"routing-overlay\""
Oct 30 12:18:30.962963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.962750    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"agc-cred-699b18e59ffa\""
Oct 30 12:18:30.970186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.970159    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"]
Oct 30 12:18:30.974964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.974939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:30.978511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.976522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:30.978828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.978808    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:30.979072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.979051    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7s6\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:30.983100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.976740    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:30.995780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.995754    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"]
Oct 30 12:18:31.008509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:30.996247    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.091478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.070914    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:18:31.091478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.071628    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:18:31.091478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.071697    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:18:31.129137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.129098    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7s6\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.129469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.129428    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.130257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.130234    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.130625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.130595    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.133810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.133680    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.134578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.134554    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.162955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.162922    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.169639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.169611    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:18:31.245587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.245535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7s6\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.262665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.262368    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:31.264740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.264713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.265077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.265055    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.265250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.265233    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkj5\" (UniqueName: \"kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.366216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.366185    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.366417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.366242    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkj5\" (UniqueName: \"kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.366417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.366315    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.367040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.367006    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.369128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.369103    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.411468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.411010    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkj5\" (UniqueName: \"kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5\") pod \"maas-api-revoke-sub-6dc8e25f-hrj7s\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.502524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.501734    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:31.822213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.822179    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"]
Oct 30 12:18:31.939915 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:18:31.939887    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e15960_72b5_4465_aa2a_04deb83f7873.slice/crio-2e7c43141777acfddfd736c8f322934988aa61f097ff271bb846a1b756e4ca0f WatchSource:0}: Error finding container 2e7c43141777acfddfd736c8f322934988aa61f097ff271bb846a1b756e4ca0f: Status 404 returned error can't find the container with id 2e7c43141777acfddfd736c8f322934988aa61f097ff271bb846a1b756e4ca0f
Oct 30 12:18:31.942358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:31.942329    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"]
Oct 30 12:18:32.373871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:32.373786    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s" event={"ID":"f6e15960-72b5-4465-aa2a-04deb83f7873","Type":"ContainerStarted","Data":"2e7c43141777acfddfd736c8f322934988aa61f097ff271bb846a1b756e4ca0f"}
Oct 30 12:18:32.375624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:32.375598    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq" event={"ID":"5f5b806c-8090-4239-9b8b-ec2caab18918","Type":"ContainerStarted","Data":"ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734"}
Oct 30 12:18:32.375778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:32.375646    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq" event={"ID":"5f5b806c-8090-4239-9b8b-ec2caab18918","Type":"ContainerStarted","Data":"015db43325ab098a866551e31545abee76c3cb3845a6104cd27cc7604e507fad"}
Oct 30 12:18:32.375778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:32.375698    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:32.429223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:32.429174    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq" podStartSLOduration=2.429160521 podStartE2EDuration="2.429160521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:32.425331209 +0000 UTC m=+2916.220215727" watchObservedRunningTime="2026-10-30 12:18:32.429160521 +0000 UTC m=+2916.224045100"
Oct 30 12:18:33.037218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:33.037184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:33.409384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:33.409286    5060 generic.go:356] "Generic (PLEG): container finished" podID="f6e15960-72b5-4465-aa2a-04deb83f7873" containerID="270bf31f4f4d5b8404d5218507968172e646fe417d6a40b70952de1356f1df75" exitCode=0
Oct 30 12:18:33.409831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:33.409371    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s" event={"ID":"f6e15960-72b5-4465-aa2a-04deb83f7873","Type":"ContainerDied","Data":"270bf31f4f4d5b8404d5218507968172e646fe417d6a40b70952de1356f1df75"}
Oct 30 12:18:34.601629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.601602    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:34.661417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.661384    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca\") pod \"f6e15960-72b5-4465-aa2a-04deb83f7873\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") "
Oct 30 12:18:34.661617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.661502    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca\") pod \"f6e15960-72b5-4465-aa2a-04deb83f7873\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") "
Oct 30 12:18:34.661705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.661627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5\" (UniqueName: \"kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5\") pod \"f6e15960-72b5-4465-aa2a-04deb83f7873\" (UID: \"f6e15960-72b5-4465-aa2a-04deb83f7873\") "
Oct 30 12:18:34.661921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.661889    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca" pod "f6e15960-72b5-4465-aa2a-04deb83f7873" (UID: "f6e15960-72b5-4465-aa2a-04deb83f7873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:34.662301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.662280    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:34.664398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.664373    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca" pod "f6e15960-72b5-4465-aa2a-04deb83f7873" (UID: "f6e15960-72b5-4465-aa2a-04deb83f7873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:18:34.664528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.664417    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5" pod "f6e15960-72b5-4465-aa2a-04deb83f7873" (UID: "f6e15960-72b5-4465-aa2a-04deb83f7873"). InnerVolumeSpecName "kube-api-access-pjkj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:34.720977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.720868    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:34.763244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.763214    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkj5\" (UniqueName: \"kubernetes.io/projected/f6e15960-72b5-4465-aa2a-04deb83f7873-kube-api-access-pjkj5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:34.763244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:34.763242    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e15960-72b5-4465-aa2a-04deb83f7873-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:35.433016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:35.432980    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"
Oct 30 12:18:35.433205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:35.433011    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s" event={"ID":"f6e15960-72b5-4465-aa2a-04deb83f7873","Type":"ContainerDied","Data":"2e7c43141777acfddfd736c8f322934988aa61f097ff271bb846a1b756e4ca0f"}
Oct 30 12:18:35.433205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:35.433064    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e7c43141777acfddfd736c8f322934988aa61f097ff271bb846a1b756e4ca0f"
Oct 30 12:18:35.459112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:35.459075    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:36.472211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:36.472010    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:36.656884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:36.656848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:36.803635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:36.803543    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:37.430638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:37.430606    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"]
Oct 30 12:18:37.433671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:37.433647    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:37.550391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:37.550361    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:38.242810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.242770    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:38.467022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.466991    5060 generic.go:356] "Generic (PLEG): container finished" podID="8abe7791-6018-475c-b11e-89f03f1206a3" containerID="ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62" exitCode=0
Oct 30 12:18:38.467198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.467061    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" event={"ID":"8abe7791-6018-475c-b11e-89f03f1206a3","Type":"ContainerDied","Data":"ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"}
Oct 30 12:18:38.467521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.467505    5060 scope.go:122] "RemoveContainer" containerID="ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"
Oct 30 12:18:38.469224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.469192    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:18:38.469324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.469264    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:18:38.469324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.469297    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:18:38.935645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:38.935547    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:39.068075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.068038    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:39.516474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.515496    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" event={"ID":"8abe7791-6018-475c-b11e-89f03f1206a3","Type":"ContainerStarted","Data":"35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314"}
Oct 30 12:18:39.543268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.538780    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"]
Oct 30 12:18:39.543268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.539802    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f6e15960-72b5-4465-aa2a-04deb83f7873" containerName="revoke-keys"
Oct 30 12:18:39.543268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.539820    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e15960-72b5-4465-aa2a-04deb83f7873" containerName="revoke-keys"
Oct 30 12:18:39.543268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.540070    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e15960-72b5-4465-aa2a-04deb83f7873" containerName="revoke-keys"
Oct 30 12:18:39.544792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.544770    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.561564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.561536    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"]
Oct 30 12:18:39.563251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.563201    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:18:39.563251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.563216    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:18:39.563251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.563206    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:18:39.563251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.563231    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:18:39.664215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.664169    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.664402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.664237    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggprg\" (UniqueName: \"kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.664402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.664361    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.765618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.765581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.765819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.765802    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.765883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.765861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggprg\" (UniqueName: \"kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.766432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.766407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.768215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.768176    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.813473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.813243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggprg\" (UniqueName: \"kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg\") pod \"maas-api-revoke-sub-1d9919bd-mdx6c\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:39.871352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:39.871313    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:40.069941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.069851    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:18:40.070490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.069982    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 12:18:40.070490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.070030    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 12:18:40.126712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.126675    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:40.215313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.215192    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"]
Oct 30 12:18:40.216307 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:18:40.216280    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee09daa_64d0_402e_aafc_a3986d1f8e82.slice/crio-a453307d4f22cfa5799bbbbb4d4b8155a7dd2e7c6069295882aba5073d3ad874 WatchSource:0}: Error finding container a453307d4f22cfa5799bbbbb4d4b8155a7dd2e7c6069295882aba5073d3ad874: Status 404 returned error can't find the container with id a453307d4f22cfa5799bbbbb4d4b8155a7dd2e7c6069295882aba5073d3ad874
Oct 30 12:18:40.527859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.527816    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c" event={"ID":"7ee09daa-64d0-402e-aafc-a3986d1f8e82","Type":"ContainerStarted","Data":"a453307d4f22cfa5799bbbbb4d4b8155a7dd2e7c6069295882aba5073d3ad874"}
Oct 30 12:18:40.611709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:40.611658    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:41.069548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:41.069519    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 12:18:41.069704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:41.069566    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 12:18:41.549811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:41.549770    5060 generic.go:356] "Generic (PLEG): container finished" podID="7ee09daa-64d0-402e-aafc-a3986d1f8e82" containerID="bcd9f4d0b328092808655af7c51cb6fdc19c96e1ecff0163340f1e6e28b77223" exitCode=0
Oct 30 12:18:41.550224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:41.549816    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c" event={"ID":"7ee09daa-64d0-402e-aafc-a3986d1f8e82","Type":"ContainerDied","Data":"bcd9f4d0b328092808655af7c51cb6fdc19c96e1ecff0163340f1e6e28b77223"}
Oct 30 12:18:41.944837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:41.944795    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:42.069789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.069755    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 12:18:42.069980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.069812    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 12:18:42.235299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.235259    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:42.410477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.401686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:42.452867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.439545    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq" podUID="5f5b806c-8090-4239-9b8b-ec2caab18918" containerName="payload-processing" containerID="cri-o://ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734" gracePeriod=25
Oct 30 12:18:42.570980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.570951    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"]
Oct 30 12:18:42.575535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.575514    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.603484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.591735    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-16c43061b500\""
Oct 30 12:18:42.603484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.591873    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-6fe65b203240\""
Oct 30 12:18:42.617591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.604573    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"kube-root-ca.crt\""
Oct 30 12:18:42.617591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.604704    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"routing-overlay\""
Oct 30 12:18:42.617591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.604796    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"openshift-service-ca.crt\""
Oct 30 12:18:42.617591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.606474    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-e2e--b2e93605-dockercfg-8bqs8\""
Oct 30 12:18:42.625712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.625682    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7b8a9d\""
Oct 30 12:18:42.631039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.630997    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"]
Oct 30 12:18:42.746246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.746210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.746592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.746332    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.746592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.746420    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.746592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.746541    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cq8n\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.890608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.870201    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.890608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.870311    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.890608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.870424    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cq8n\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.890608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.870530    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.890608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.871393    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.937480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.930014    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.955476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.953875    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"]
Oct 30 12:18:42.993784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.993752    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:42.997435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.997117    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:42.998933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.998818    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"]
Oct 30 12:18:42.999103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:42.998910    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.085840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.082421    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 12:18:43.090538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.082493    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 12:18:43.094908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.094881    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg\" (UniqueName: \"kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg\") pod \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") "
Oct 30 12:18:43.095078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.095063    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca\") pod \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") "
Oct 30 12:18:43.095242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.095225    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca\") pod \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\" (UID: \"7ee09daa-64d0-402e-aafc-a3986d1f8e82\") "
Oct 30 12:18:43.095944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.095920    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfx6\" (UniqueName: \"kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.096233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.096212    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.096429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.096411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.099692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.099531    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg" pod "7ee09daa-64d0-402e-aafc-a3986d1f8e82" (UID: "7ee09daa-64d0-402e-aafc-a3986d1f8e82"). InnerVolumeSpecName "kube-api-access-ggprg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:43.100253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.100000    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca" pod "7ee09daa-64d0-402e-aafc-a3986d1f8e82" (UID: "7ee09daa-64d0-402e-aafc-a3986d1f8e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:43.104241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.102762    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca" pod "7ee09daa-64d0-402e-aafc-a3986d1f8e82" (UID: "7ee09daa-64d0-402e-aafc-a3986d1f8e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:18:43.130399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.129803    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cq8n\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:43.144527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.142654    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.215793    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfx6\" (UniqueName: \"kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.215987    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.216092    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.216242    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggprg\" (UniqueName: \"kubernetes.io/projected/7ee09daa-64d0-402e-aafc-a3986d1f8e82-kube-api-access-ggprg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.216259    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.216273    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee09daa-64d0-402e-aafc-a3986d1f8e82-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.221473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.219567    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.225825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.224322    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.227481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.227397    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:43.242175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.242155    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfx6\" (UniqueName: \"kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6\") pod \"maas-api-revoke-sub-67f95da1-vhtsl\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.317388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.317348    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6\") pod \"5f5b806c-8090-4239-9b8b-ec2caab18918\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") "
Oct 30 12:18:43.320518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.320488    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume\") pod \"5f5b806c-8090-4239-9b8b-ec2caab18918\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") "
Oct 30 12:18:43.320780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.320759    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials\") pod \"5f5b806c-8090-4239-9b8b-ec2caab18918\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") "
Oct 30 12:18:43.320933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.320917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume\") pod \"5f5b806c-8090-4239-9b8b-ec2caab18918\" (UID: \"5f5b806c-8090-4239-9b8b-ec2caab18918\") "
Oct 30 12:18:43.321751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.321296    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume" pod "5f5b806c-8090-4239-9b8b-ec2caab18918" (UID: "5f5b806c-8090-4239-9b8b-ec2caab18918"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:43.321751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.321724    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.322181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.322158    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume" pod "5f5b806c-8090-4239-9b8b-ec2caab18918" (UID: "5f5b806c-8090-4239-9b8b-ec2caab18918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:43.323697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.323660    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6" pod "5f5b806c-8090-4239-9b8b-ec2caab18918" (UID: "5f5b806c-8090-4239-9b8b-ec2caab18918"). InnerVolumeSpecName "kube-api-access-qv7s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:43.324629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.324602    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials" pod "5f5b806c-8090-4239-9b8b-ec2caab18918" (UID: "5f5b806c-8090-4239-9b8b-ec2caab18918"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:43.425912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.425816    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5b806c-8090-4239-9b8b-ec2caab18918-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.425912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.425854    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.425912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.425871    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv7s6\" (UniqueName: \"kubernetes.io/projected/5f5b806c-8090-4239-9b8b-ec2caab18918-kube-api-access-qv7s6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:43.431711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.431686    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:43.566031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.565703    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"
Oct 30 12:18:43.566031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.565712    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c" event={"ID":"7ee09daa-64d0-402e-aafc-a3986d1f8e82","Type":"ContainerDied","Data":"a453307d4f22cfa5799bbbbb4d4b8155a7dd2e7c6069295882aba5073d3ad874"}
Oct 30 12:18:43.566031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.565766    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a453307d4f22cfa5799bbbbb4d4b8155a7dd2e7c6069295882aba5073d3ad874"
Oct 30 12:18:43.567336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.567311    5060 generic.go:356] "Generic (PLEG): container finished" podID="5f5b806c-8090-4239-9b8b-ec2caab18918" containerID="ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734" exitCode=0
Oct 30 12:18:43.567441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.567360    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"
Oct 30 12:18:43.567441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.567392    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq" event={"ID":"5f5b806c-8090-4239-9b8b-ec2caab18918","Type":"ContainerDied","Data":"ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734"}
Oct 30 12:18:43.567441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.567436    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq" event={"ID":"5f5b806c-8090-4239-9b8b-ec2caab18918","Type":"ContainerDied","Data":"015db43325ab098a866551e31545abee76c3cb3845a6104cd27cc7604e507fad"}
Oct 30 12:18:43.567601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.567486    5060 scope.go:122] "RemoveContainer" containerID="ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734"
Oct 30 12:18:43.623505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.623283    5060 scope.go:122] "RemoveContainer" containerID="ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734"
Oct 30 12:18:43.626028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.625999    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"]
Oct 30 12:18:43.633720 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:18:43.633477    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734\": container with ID starting with ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734 not found: ID does not exist" containerID="ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734"
Oct 30 12:18:43.633720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.633532    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734"} err="failed to get container status \"ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734\": rpc error: code = NotFound desc = could not find container \"ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734\": container with ID starting with ee3aac16decccc615d9726b132004c043ee03f7a01996e81979106f6a3c75734 not found: ID does not exist"
Oct 30 12:18:43.655546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.655510    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"]
Oct 30 12:18:43.658746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.658723    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-5cffftsgq"]
Oct 30 12:18:43.726549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.726523    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:43.828871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:43.825351    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"]
Oct 30 12:18:43.833057 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:18:43.833027    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7f05bb_b672_486d_8870_1eeeb1dbdc71.slice/crio-bdd42a711d8945f99762e63964d219217fa9b2cbe9c91cce74ec32dd033009e5 WatchSource:0}: Error finding container bdd42a711d8945f99762e63964d219217fa9b2cbe9c91cce74ec32dd033009e5: Status 404 returned error can't find the container with id bdd42a711d8945f99762e63964d219217fa9b2cbe9c91cce74ec32dd033009e5
Oct 30 12:18:44.069591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.069557    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 12:18:44.069771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.069610    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 12:18:44.336010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.335982    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5b806c-8090-4239-9b8b-ec2caab18918" path="/var/lib/kubelet/pods/5f5b806c-8090-4239-9b8b-ec2caab18918/volumes"
Oct 30 12:18:44.615502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.615354    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl" event={"ID":"ea7f05bb-b672-486d-8870-1eeeb1dbdc71","Type":"ContainerStarted","Data":"8e73435d8b67b2b5f8b100b4949282c29ad8b6a4bc7fb418377385f90d1c857b"}
Oct 30 12:18:44.615784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.615760    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl" event={"ID":"ea7f05bb-b672-486d-8870-1eeeb1dbdc71","Type":"ContainerStarted","Data":"bdd42a711d8945f99762e63964d219217fa9b2cbe9c91cce74ec32dd033009e5"}
Oct 30 12:18:44.619310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.619282    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9" event={"ID":"f5920099-df86-4b5a-84ad-493bb40aa5dc","Type":"ContainerStarted","Data":"85ae120f770213a9c7cbf259ccd7bfe5c6534230eb9d0930c448f0d420a61030"}
Oct 30 12:18:44.619432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.619336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9" event={"ID":"f5920099-df86-4b5a-84ad-493bb40aa5dc","Type":"ContainerStarted","Data":"0139ded675956357662d1a63db334499187a8660fb18422ac0582f6bf59e3a25"}
Oct 30 12:18:44.619432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.619369    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:44.658363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.658312    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl" podStartSLOduration=2.658297271 podStartE2EDuration="2.658297271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:44.657031663 +0000 UTC m=+2928.451916157" watchObservedRunningTime="2026-10-30 12:18:44.658297271 +0000 UTC m=+2928.453181765"
Oct 30 12:18:44.717482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:44.700630    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9" podStartSLOduration=2.70060816 podStartE2EDuration="2.70060816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:44.697962673 +0000 UTC m=+2928.492847170" watchObservedRunningTime="2026-10-30 12:18:44.70060816 +0000 UTC m=+2928.495492656"
Oct 30 12:18:45.129617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.129584    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:18:45.129981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.129953    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:18:45.130755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.130733    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:18:45.560925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.560895    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:45.628884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.628853    5060 generic.go:356] "Generic (PLEG): container finished" podID="ea7f05bb-b672-486d-8870-1eeeb1dbdc71" containerID="8e73435d8b67b2b5f8b100b4949282c29ad8b6a4bc7fb418377385f90d1c857b" exitCode=0
Oct 30 12:18:45.629091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.628947    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl" event={"ID":"ea7f05bb-b672-486d-8870-1eeeb1dbdc71","Type":"ContainerDied","Data":"8e73435d8b67b2b5f8b100b4949282c29ad8b6a4bc7fb418377385f90d1c857b"}
Oct 30 12:18:45.823766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.823662    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:45.961090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:45.961050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:46.478912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.478875    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:46.630323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.630262    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:46.778283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.777916    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:46.804011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.803986    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:46.879675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.879642    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca\") pod \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") "
Oct 30 12:18:46.880172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.879784    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6\" (UniqueName: \"kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6\") pod \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") "
Oct 30 12:18:46.880172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.879938    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca\") pod \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\" (UID: \"ea7f05bb-b672-486d-8870-1eeeb1dbdc71\") "
Oct 30 12:18:46.880172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.880017    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca" pod "ea7f05bb-b672-486d-8870-1eeeb1dbdc71" (UID: "ea7f05bb-b672-486d-8870-1eeeb1dbdc71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:46.880556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.880529    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:46.882248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.882214    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6" pod "ea7f05bb-b672-486d-8870-1eeeb1dbdc71" (UID: "ea7f05bb-b672-486d-8870-1eeeb1dbdc71"). InnerVolumeSpecName "kube-api-access-bcfx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:46.882650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.882629    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca" pod "ea7f05bb-b672-486d-8870-1eeeb1dbdc71" (UID: "ea7f05bb-b672-486d-8870-1eeeb1dbdc71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:18:46.981626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.981593    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:46.981626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.981623    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcfx6\" (UniqueName: \"kubernetes.io/projected/ea7f05bb-b672-486d-8870-1eeeb1dbdc71-kube-api-access-bcfx6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:46.994398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:46.994357    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:47.328863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.328827    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.353629    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"]
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.354706    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ea7f05bb-b672-486d-8870-1eeeb1dbdc71" containerName="revoke-keys"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.354726    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7f05bb-b672-486d-8870-1eeeb1dbdc71" containerName="revoke-keys"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356166    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ee09daa-64d0-402e-aafc-a3986d1f8e82" containerName="revoke-keys"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356199    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee09daa-64d0-402e-aafc-a3986d1f8e82" containerName="revoke-keys"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356271    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f5b806c-8090-4239-9b8b-ec2caab18918" containerName="payload-processing"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356280    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5b806c-8090-4239-9b8b-ec2caab18918" containerName="payload-processing"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356506    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7f05bb-b672-486d-8870-1eeeb1dbdc71" containerName="revoke-keys"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356531    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee09daa-64d0-402e-aafc-a3986d1f8e82" containerName="revoke-keys"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.356558    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5b806c-8090-4239-9b8b-ec2caab18918" containerName="payload-processing"
Oct 30 12:18:47.371480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.366530    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.410540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.408527    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"]
Oct 30 12:18:47.625339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.625240    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zwb\" (UniqueName: \"kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.625708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.625681    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.625972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.625955    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.730683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.730638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:47.735049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.735017    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zwb\" (UniqueName: \"kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.735366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.735347    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.735620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.735599    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.736773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.736749    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.752270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.739411    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.752270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.743594    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl" event={"ID":"ea7f05bb-b672-486d-8870-1eeeb1dbdc71","Type":"ContainerDied","Data":"bdd42a711d8945f99762e63964d219217fa9b2cbe9c91cce74ec32dd033009e5"}
Oct 30 12:18:47.752270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.743647    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd42a711d8945f99762e63964d219217fa9b2cbe9c91cce74ec32dd033009e5"
Oct 30 12:18:47.752270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.743715    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"
Oct 30 12:18:47.769009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.768654    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zwb\" (UniqueName: \"kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb\") pod \"maas-api-revoke-sub-eedc5ba7-dlwql\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.774697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.773511    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:47.881830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:47.881704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:48.123118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:48.123066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:48.144346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:48.144316    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"]
Oct 30 12:18:48.263316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:48.263265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:48.758826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:48.758790    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql" event={"ID":"13430e6c-b863-43ce-9459-fb4042803794","Type":"ContainerStarted","Data":"8fd81a5779dab9d57750025bb484ef585308ac958c30576764897c46ab892f2f"}
Oct 30 12:18:48.759002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:48.758865    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql" event={"ID":"13430e6c-b863-43ce-9459-fb4042803794","Type":"ContainerStarted","Data":"44f4475908a2d13e73574b9004c0f65e4c4e7083dfdf073e455ce49bdb022ba4"}
Oct 30 12:18:48.805309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:48.805259    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql" podStartSLOduration=1.80524536 podStartE2EDuration="1.80524536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:48.803124269 +0000 UTC m=+2932.598008763" watchObservedRunningTime="2026-10-30 12:18:48.80524536 +0000 UTC m=+2932.600129853"
Oct 30 12:18:49.140990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:49.140885    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:49.282147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:49.282110    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:49.549814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:49.549779    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:49.689306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:49.689260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:49.770050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:49.770012    5060 generic.go:356] "Generic (PLEG): container finished" podID="13430e6c-b863-43ce-9459-fb4042803794" containerID="8fd81a5779dab9d57750025bb484ef585308ac958c30576764897c46ab892f2f" exitCode=0
Oct 30 12:18:49.770282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:49.770252    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql" event={"ID":"13430e6c-b863-43ce-9459-fb4042803794","Type":"ContainerDied","Data":"8fd81a5779dab9d57750025bb484ef585308ac958c30576764897c46ab892f2f"}
Oct 30 12:18:49.861004 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:18:49.860903    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:18:50.914220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:50.914187    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:51.027432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.027406    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:51.065530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.065385    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:51.101470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.087028    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb\" (UniqueName: \"kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb\") pod \"13430e6c-b863-43ce-9459-fb4042803794\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") "
Oct 30 12:18:51.101470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.087162    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca\") pod \"13430e6c-b863-43ce-9459-fb4042803794\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") "
Oct 30 12:18:51.101470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.087261    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca\") pod \"13430e6c-b863-43ce-9459-fb4042803794\" (UID: \"13430e6c-b863-43ce-9459-fb4042803794\") "
Oct 30 12:18:51.101470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.088800    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca" pod "13430e6c-b863-43ce-9459-fb4042803794" (UID: "13430e6c-b863-43ce-9459-fb4042803794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:51.101470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.093000    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca" pod "13430e6c-b863-43ce-9459-fb4042803794" (UID: "13430e6c-b863-43ce-9459-fb4042803794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:18:51.101470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.095856    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb" pod "13430e6c-b863-43ce-9459-fb4042803794" (UID: "13430e6c-b863-43ce-9459-fb4042803794"). InnerVolumeSpecName "kube-api-access-z4zwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:51.189371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.189324    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13430e6c-b863-43ce-9459-fb4042803794-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:51.189371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.189361    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zwb\" (UniqueName: \"kubernetes.io/projected/13430e6c-b863-43ce-9459-fb4042803794-kube-api-access-z4zwb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:51.189371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.189377    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13430e6c-b863-43ce-9459-fb4042803794-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:51.209253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.209215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:51.359303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.359212    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:51.790615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.790579    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql" event={"ID":"13430e6c-b863-43ce-9459-fb4042803794","Type":"ContainerDied","Data":"44f4475908a2d13e73574b9004c0f65e4c4e7083dfdf073e455ce49bdb022ba4"}
Oct 30 12:18:51.790802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.790637    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f4475908a2d13e73574b9004c0f65e4c4e7083dfdf073e455ce49bdb022ba4"
Oct 30 12:18:51.790802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:51.790705    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"
Oct 30 12:18:52.139040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:52.138923    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:52.279908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:52.279863    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:52.680818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:52.680780    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:52.823161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:52.823128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:52.982304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:52.982271    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:53.227596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:53.227561    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:53.371343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:53.371253    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:54.296162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:54.296126    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:54.440790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:54.440753    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:54.650647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:54.650551    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"]
Oct 30 12:18:54.653770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:54.653747    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:18:55.030989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.030950    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"]
Oct 30 12:18:55.031716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.031695    5060 cpu_manager.go:414] "containerMap: removing container" podUID="13430e6c-b863-43ce-9459-fb4042803794" containerName="revoke-keys"
Oct 30 12:18:55.031716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.031717    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13430e6c-b863-43ce-9459-fb4042803794" containerName="revoke-keys"
Oct 30 12:18:55.031927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.031913    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="13430e6c-b863-43ce-9459-fb4042803794" containerName="revoke-keys"
Oct 30 12:18:55.035601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.035580    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.050933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.050912    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:18:55.053243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.053220    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"]
Oct 30 12:18:55.060587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.060557    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:18:55.060715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.060603    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:18:55.060800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.060732    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:18:55.078155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.078122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.078313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.078175    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b7j\" (UniqueName: \"kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.078486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.078436    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.179641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.179609    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.179851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.179713    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.179851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.179737    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b7j\" (UniqueName: \"kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.180435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.180413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.182176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.182152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.204845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.204814    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b7j\" (UniqueName: \"kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j\") pod \"maas-api-revoke-sub-f9184d92-4k6d2\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.251576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.251547    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:55.350853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.350777    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:55.660691 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:18:55.660663    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eab30c0_09cd_4b8b_ab48_36427b5fbf7b.slice/crio-ce489193b9a77ed30a081d3ad47148733fb75c671bc14afe32776915f4317b08 WatchSource:0}: Error finding container ce489193b9a77ed30a081d3ad47148733fb75c671bc14afe32776915f4317b08: Status 404 returned error can't find the container with id ce489193b9a77ed30a081d3ad47148733fb75c671bc14afe32776915f4317b08
Oct 30 12:18:55.662228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.662197    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"]
Oct 30 12:18:55.831493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:55.831444    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2" event={"ID":"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b","Type":"ContainerStarted","Data":"ce489193b9a77ed30a081d3ad47148733fb75c671bc14afe32776915f4317b08"}
Oct 30 12:18:56.506475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:56.506424    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:56.653677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:56.653640    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:56.842092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:56.841999    5060 generic.go:356] "Generic (PLEG): container finished" podID="5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" containerID="f3ef4ad8a351e0b458e088abf2ff4d59c219310c991de14b46ae5ab877192892" exitCode=0
Oct 30 12:18:56.842092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:56.842049    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2" event={"ID":"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b","Type":"ContainerDied","Data":"f3ef4ad8a351e0b458e088abf2ff4d59c219310c991de14b46ae5ab877192892"}
Oct 30 12:18:56.871820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:56.871784    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:57.056347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:57.056307    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:58.037699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.037674    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:58.128001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.127919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j\" (UniqueName: \"kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j\") pod \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") "
Oct 30 12:18:58.128163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.128102    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca\") pod \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") "
Oct 30 12:18:58.128163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.128157    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca\") pod \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\" (UID: \"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b\") "
Oct 30 12:18:58.128571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.128543    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca" pod "5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" (UID: "5eab30c0-09cd-4b8b-ab48-36427b5fbf7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:58.128732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.128715    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:58.130976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.130946    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j" pod "5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" (UID: "5eab30c0-09cd-4b8b-ab48-36427b5fbf7b"). InnerVolumeSpecName "kube-api-access-z6b7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:58.131095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.131003    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca" pod "5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" (UID: "5eab30c0-09cd-4b8b-ab48-36427b5fbf7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:18:58.229541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.229515    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b7j\" (UniqueName: \"kubernetes.io/projected/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-kube-api-access-z6b7j\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:58.229541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.229539    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:18:58.689573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.689539    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:58.843477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.840029    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:58.909505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.905504    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2" event={"ID":"5eab30c0-09cd-4b8b-ab48-36427b5fbf7b","Type":"ContainerDied","Data":"ce489193b9a77ed30a081d3ad47148733fb75c671bc14afe32776915f4317b08"}
Oct 30 12:18:58.909505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.905586    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce489193b9a77ed30a081d3ad47148733fb75c671bc14afe32776915f4317b08"
Oct 30 12:18:58.909505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:58.905691    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"
Oct 30 12:18:59.096798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.096759    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:59.323474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.323370    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:59.486489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.485987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:59.631726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.631697    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:18:59.651168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.651131    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9" podUID="f5920099-df86-4b5a-84ad-493bb40aa5dc" containerName="payload-processing" containerID="cri-o://85ae120f770213a9c7cbf259ccd7bfe5c6534230eb9d0930c448f0d420a61030" gracePeriod=25
Oct 30 12:18:59.918693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.918665    5060 generic.go:356] "Generic (PLEG): container finished" podID="f5920099-df86-4b5a-84ad-493bb40aa5dc" containerID="85ae120f770213a9c7cbf259ccd7bfe5c6534230eb9d0930c448f0d420a61030" exitCode=0
Oct 30 12:18:59.918807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.918736    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9" event={"ID":"f5920099-df86-4b5a-84ad-493bb40aa5dc","Type":"ContainerDied","Data":"85ae120f770213a9c7cbf259ccd7bfe5c6534230eb9d0930c448f0d420a61030"}
Oct 30 12:18:59.947821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:18:59.947798    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:19:00.084988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.084861    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials\") pod \"f5920099-df86-4b5a-84ad-493bb40aa5dc\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") "
Oct 30 12:19:00.084988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.084988    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume\") pod \"f5920099-df86-4b5a-84ad-493bb40aa5dc\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") "
Oct 30 12:19:00.085235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.085024    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n\") pod \"f5920099-df86-4b5a-84ad-493bb40aa5dc\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") "
Oct 30 12:19:00.085235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.085144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume\") pod \"f5920099-df86-4b5a-84ad-493bb40aa5dc\" (UID: \"f5920099-df86-4b5a-84ad-493bb40aa5dc\") "
Oct 30 12:19:00.085437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.085409    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume" pod "f5920099-df86-4b5a-84ad-493bb40aa5dc" (UID: "f5920099-df86-4b5a-84ad-493bb40aa5dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:00.085634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.085521    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume" pod "f5920099-df86-4b5a-84ad-493bb40aa5dc" (UID: "f5920099-df86-4b5a-84ad-493bb40aa5dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:00.086077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.086059    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:00.086180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.086079    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920099-df86-4b5a-84ad-493bb40aa5dc-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:00.087781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.087731    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials" pod "f5920099-df86-4b5a-84ad-493bb40aa5dc" (UID: "f5920099-df86-4b5a-84ad-493bb40aa5dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:00.087890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.087803    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n" pod "f5920099-df86-4b5a-84ad-493bb40aa5dc" (UID: "f5920099-df86-4b5a-84ad-493bb40aa5dc"). InnerVolumeSpecName "kube-api-access-2cq8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:00.187396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.187359    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:00.187396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.187391    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cq8n\" (UniqueName: \"kubernetes.io/projected/f5920099-df86-4b5a-84ad-493bb40aa5dc-kube-api-access-2cq8n\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:00.600510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.600392    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:00.930921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.930822    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"
Oct 30 12:19:00.930921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.930836    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9" event={"ID":"f5920099-df86-4b5a-84ad-493bb40aa5dc","Type":"ContainerDied","Data":"0139ded675956357662d1a63db334499187a8660fb18422ac0582f6bf59e3a25"}
Oct 30 12:19:00.930921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.930915    5060 scope.go:122] "RemoveContainer" containerID="85ae120f770213a9c7cbf259ccd7bfe5c6534230eb9d0930c448f0d420a61030"
Oct 30 12:19:00.985208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.985173    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"]
Oct 30 12:19:00.999471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:00.999422    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667zcvv9"]
Oct 30 12:19:02.336955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:02.336919    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5920099-df86-4b5a-84ad-493bb40aa5dc" path="/var/lib/kubelet/pods/f5920099-df86-4b5a-84ad-493bb40aa5dc/volumes"
Oct 30 12:19:02.355692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:02.355665    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:03.066620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:03.066586    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:03.224567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:03.224533    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:03.567037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:03.566999    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:03.759996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:03.759967    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:04.337993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.337944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:04.593530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.593408    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:04.782406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.782365    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"]
Oct 30 12:19:04.783390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.783367    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" containerName="revoke-keys"
Oct 30 12:19:04.783539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.783391    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" containerName="revoke-keys"
Oct 30 12:19:04.783539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.783443    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f5920099-df86-4b5a-84ad-493bb40aa5dc" containerName="payload-processing"
Oct 30 12:19:04.783539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.783469    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5920099-df86-4b5a-84ad-493bb40aa5dc" containerName="payload-processing"
Oct 30 12:19:04.783770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.783624    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" containerName="revoke-keys"
Oct 30 12:19:04.783770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.783642    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5920099-df86-4b5a-84ad-493bb40aa5dc" containerName="payload-processing"
Oct 30 12:19:04.788238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.788215    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.805652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.805625    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"]
Oct 30 12:19:04.811100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.811074    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-67e819\"/\"payload-processing-external-model-plugins-e2e-extmod-a-67e819\""
Oct 30 12:19:04.811300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.811100    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-67e819\"/\"e2e-cmp-legacy-d1daec-provider-cred\""
Oct 30 12:19:04.813042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.811299    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-67e819\"/\"kube-root-ca.crt\""
Oct 30 12:19:04.813141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.811502    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-67e819\"/\"routing-overlay\""
Oct 30 12:19:04.813219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.811556    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-67e819\"/\"openshift-service-ca.crt\""
Oct 30 12:19:04.815836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.815817    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-67e819\"/\"payload-processing-external-model-e2e--21e0f139-dockercfg-5ncp2\""
Oct 30 12:19:04.859000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.858907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6c9g\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.859000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.858975    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.859217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.859048    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.859217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.859096    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.960824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.960622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6c9g\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.960824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.960712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.960824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.960753    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.984876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.960845    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.984876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.964111    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:04.984876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:04.982393    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:05.015796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:05.001159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:05.044475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:05.044076    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6c9g\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g\") pod \"payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") " pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:05.112829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:05.112734    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:05.456717 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:05.456687    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9f1251_c3d0_4545_993d_ba423a6e1264.slice/crio-12eb1458a90099177d3343afbe6cbf6a236d88615662f5aef5855037b24b74a4 WatchSource:0}: Error finding container 12eb1458a90099177d3343afbe6cbf6a236d88615662f5aef5855037b24b74a4: Status 404 returned error can't find the container with id 12eb1458a90099177d3343afbe6cbf6a236d88615662f5aef5855037b24b74a4
Oct 30 12:19:05.458718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:05.458691    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"]
Oct 30 12:19:06.050718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.050683    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl" event={"ID":"7f9f1251-c3d0-4545-993d-ba423a6e1264","Type":"ContainerStarted","Data":"1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32"}
Oct 30 12:19:06.051172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.050738    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl" event={"ID":"7f9f1251-c3d0-4545-993d-ba423a6e1264","Type":"ContainerStarted","Data":"12eb1458a90099177d3343afbe6cbf6a236d88615662f5aef5855037b24b74a4"}
Oct 30 12:19:06.051172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.050768    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:06.092211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.092147    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl" podStartSLOduration=2.092128508 podStartE2EDuration="2.092128508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:06.091184369 +0000 UTC m=+2949.886068867" watchObservedRunningTime="2026-10-30 12:19:06.092128508 +0000 UTC m=+2949.887013003"
Oct 30 12:19:06.145473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.145415    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:06.307407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.307311    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:06.465059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.465015    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:06.606011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.605913    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:06.915048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:06.914952    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:07.089823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:07.089782    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:07.246356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:07.246320    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:07.392998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:07.392964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:08.145305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:08.145267    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:08.750802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:08.750765    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:10.175772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:10.175738    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:11.849902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:11.849861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:13.398686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:13.398630    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:13.543412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:13.543378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:13.686396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:13.686298    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:13.830170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:13.830132    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:13.979044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:13.979007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:14.129144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:14.129113    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:14.378783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:14.378682    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:14.526760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:14.526727    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:15.656199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:15.656152    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:17.130759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:17.130730    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:19:17.463745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:17.463685    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:19.313368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:19.313327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:19.858627 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:19.858591    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:19:20.283912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:20.283871    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:20.437722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:20.437686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:20.590084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:20.589991    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:20.757853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:20.757817    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:20.913259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:20.913172    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:21.058774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:21.058724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:21.206368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:21.206332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:21.351336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:21.351301    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:23.215343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:23.215307    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:24.932191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:24.932144    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:26.607923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:26.607884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:28.368134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:28.368083    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:30.047072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:30.047029    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:31.844823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:31.844785    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:32.008477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:32.008192    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:33.040975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.040942    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:33.191768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.191731    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:33.356973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.356884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:33.484903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.484861    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"]
Oct 30 12:19:33.519680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.519641    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"]
Oct 30 12:19:33.520546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.520471    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.552331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.552301    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"openshift-service-ca.crt\""
Oct 30 12:19:33.552588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.552560    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"payload-processing-external-model-e2e--f69e4050-dockercfg-s2k9p\""
Oct 30 12:19:33.552723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.552467    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"routing-overlay\""
Oct 30 12:19:33.552723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.552506    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"agc-cred-93e88bcdc5ff\""
Oct 30 12:19:33.552723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.552532    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"payload-processing-external-model-plugins-e2e-extmod-a-737f64\""
Oct 30 12:19:33.552723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.552393    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"kube-root-ca.crt\""
Oct 30 12:19:33.717516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.708362    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.717516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.708480    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.717516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.708622    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfjr\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.717516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.708665    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.810328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.810287    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.810635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.810393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.810635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.810565    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfjr\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.810635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.810612    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.811137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.811108    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.811544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.811518    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.813570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.813544    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.834306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.834282    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfjr\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:33.847175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:33.847150    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:34.000205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.000133    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"]
Oct 30 12:19:34.004789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.004769    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.019743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.019707    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"agc-cred-caefd8249446\""
Oct 30 12:19:34.024128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.024090    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"]
Oct 30 12:19:34.115083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.115045    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.115558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.115251    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9t4c\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.115558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.115406    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.115558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.115489    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.186769 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:34.186741    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51bc15e_9915_4720_be99_f383c4d72f53.slice/crio-90437d486bb326326c44016fb1f782d45c4e3649739c2026a30eb9adf0466d21 WatchSource:0}: Error finding container 90437d486bb326326c44016fb1f782d45c4e3649739c2026a30eb9adf0466d21: Status 404 returned error can't find the container with id 90437d486bb326326c44016fb1f782d45c4e3649739c2026a30eb9adf0466d21
Oct 30 12:19:34.188360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.188337    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"]
Oct 30 12:19:34.218107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.217138    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.218107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.217240    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9t4c\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.218107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.217320    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.218107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.217352    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.218107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.217913    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.218107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.218064    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.220605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.220584    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.255648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.255613    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9t4c\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.299716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.299677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:34.321049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.321019    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:34.429037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.428999    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5" event={"ID":"e51bc15e-9915-4720-be99-f383c4d72f53","Type":"ContainerStarted","Data":"b664ff81dd1c4f4cc3ae517027767003520b14c347b00a42a20745e6648bbfa4"}
Oct 30 12:19:34.429245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.429065    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5" event={"ID":"e51bc15e-9915-4720-be99-f383c4d72f53","Type":"ContainerStarted","Data":"90437d486bb326326c44016fb1f782d45c4e3649739c2026a30eb9adf0466d21"}
Oct 30 12:19:34.429245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.429093    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:34.475171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.475121    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5" podStartSLOduration=1.47510597 podStartE2EDuration="1.47510597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:34.47064714 +0000 UTC m=+2978.265531634" watchObservedRunningTime="2026-10-30 12:19:34.47510597 +0000 UTC m=+2978.269990488"
Oct 30 12:19:34.636188 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:34.636161    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cf6517_b8f4_4324_b9ca_07308b9e1297.slice/crio-88993f2413ccfe637b97fa0afcf5ace3d92374203aa43a4f2a88908240abca69 WatchSource:0}: Error finding container 88993f2413ccfe637b97fa0afcf5ace3d92374203aa43a4f2a88908240abca69: Status 404 returned error can't find the container with id 88993f2413ccfe637b97fa0afcf5ace3d92374203aa43a4f2a88908240abca69
Oct 30 12:19:34.638585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:34.638558    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"]
Oct 30 12:19:35.439178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:35.439142    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm" event={"ID":"08cf6517-b8f4-4324-b9ca-07308b9e1297","Type":"ContainerStarted","Data":"37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b"}
Oct 30 12:19:35.439658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:35.439203    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm" event={"ID":"08cf6517-b8f4-4324-b9ca-07308b9e1297","Type":"ContainerStarted","Data":"88993f2413ccfe637b97fa0afcf5ace3d92374203aa43a4f2a88908240abca69"}
Oct 30 12:19:35.439658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:35.439432    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:35.487187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:35.487130    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm" podStartSLOduration=2.4871153550000002 podStartE2EDuration="2.487115355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:35.482732452 +0000 UTC m=+2979.277616945" watchObservedRunningTime="2026-10-30 12:19:35.487115355 +0000 UTC m=+2979.281999849"
Oct 30 12:19:36.166317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:36.166278    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:37.950145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:37.950103    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:39.946489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:39.946403    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:41.828726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:41.828688    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:42.391348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.391312    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"]
Oct 30 12:19:42.397013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.396997    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.413783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.413755    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"]
Oct 30 12:19:42.420069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.419977    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:19:42.420069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.419982    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:19:42.420069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.419981    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:19:42.424098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.424075    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:19:42.462641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.462609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.462840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.462771    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.462936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.462916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj92m\" (UniqueName: \"kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.564026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.563989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.564237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.564134    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj92m\" (UniqueName: \"kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.564237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.564212    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.564910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.564839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.566565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.566543    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.588035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.588006    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj92m\" (UniqueName: \"kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m\") pod \"maas-api-revoke-sub-e3e97f78-hsnmc\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:42.714186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:42.714152    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:43.056703 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:43.056677    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9781c203_4f34_4803_98a4_ea446780d716.slice/crio-4d4658ca3cc0cced31e4f166e90c8a588dabd1cdfd2e791d4407507b902854be WatchSource:0}: Error finding container 4d4658ca3cc0cced31e4f166e90c8a588dabd1cdfd2e791d4407507b902854be: Status 404 returned error can't find the container with id 4d4658ca3cc0cced31e4f166e90c8a588dabd1cdfd2e791d4407507b902854be
Oct 30 12:19:43.059646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.059620    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"]
Oct 30 12:19:43.279848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.279822    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"]
Oct 30 12:19:43.285921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.285901    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.304819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.304745    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"]
Oct 30 12:19:43.311867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.311843    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"]
Oct 30 12:19:43.317593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.317572    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.333535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.333470    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"]
Oct 30 12:19:43.374893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.374853    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.375259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.375196    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6jf\" (UniqueName: \"kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.375425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.375400    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.375542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.375444    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.375542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.375515    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.375645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.375627    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86cx\" (UniqueName: \"kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.476924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.476882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.477145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.476940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.477145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.476987    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.477145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.477053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86cx\" (UniqueName: \"kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.477323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.477184    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.477323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.477294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6jf\" (UniqueName: \"kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.477976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.477950    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.478531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.478505    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.480205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.480181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.480343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.480275    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.501885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.501855    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86cx\" (UniqueName: \"kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx\") pod \"maas-api-revoke-sub-7ce9c3b4-pnx8w\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.502081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.501882    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6jf\" (UniqueName: \"kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf\") pod \"maas-api-revoke-sub-76fe7c53-96jbs\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.612262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.612176    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:43.637687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.637655    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:43.661150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.661118    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc" event={"ID":"9781c203-4f34-4803-98a4-ea446780d716","Type":"ContainerStarted","Data":"95001cac3ed4679616415505c5eda485e32364235a8fb515f54dc995688d1905"}
Oct 30 12:19:43.661299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.661182    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc" event={"ID":"9781c203-4f34-4803-98a4-ea446780d716","Type":"ContainerStarted","Data":"4d4658ca3cc0cced31e4f166e90c8a588dabd1cdfd2e791d4407507b902854be"}
Oct 30 12:19:43.709756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.709695    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc" podStartSLOduration=1.709675312 podStartE2EDuration="1.709675312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:43.707050803 +0000 UTC m=+2987.501935309" watchObservedRunningTime="2026-10-30 12:19:43.709675312 +0000 UTC m=+2987.504559807"
Oct 30 12:19:43.852042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.852008    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:43.931862 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:43.931729    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eadd852_c051_4788_a118_9cc91bc2c1ce.slice/crio-a46ec595cf413a9e2337606f4e608f12bcde2fee21658c95071245d3c2641b92 WatchSource:0}: Error finding container a46ec595cf413a9e2337606f4e608f12bcde2fee21658c95071245d3c2641b92: Status 404 returned error can't find the container with id a46ec595cf413a9e2337606f4e608f12bcde2fee21658c95071245d3c2641b92
Oct 30 12:19:43.933919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.932697    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"]
Oct 30 12:19:43.965038 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:43.965007    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae53798_5b70_4ad1_a0b1_b8d6995fbb52.slice/crio-560affbf0fbc2b39b6fc6214aac480cb319f5eb071ad12bbb9f4cd8848293823 WatchSource:0}: Error finding container 560affbf0fbc2b39b6fc6214aac480cb319f5eb071ad12bbb9f4cd8848293823: Status 404 returned error can't find the container with id 560affbf0fbc2b39b6fc6214aac480cb319f5eb071ad12bbb9f4cd8848293823
Oct 30 12:19:43.965395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:43.965371    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"]
Oct 30 12:19:44.670465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.670415    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w" event={"ID":"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52","Type":"ContainerStarted","Data":"210945bdd03526d5911a09ed503443971fa92b38768b48b54212d484f6fda5e5"}
Oct 30 12:19:44.670928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.670482    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w" event={"ID":"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52","Type":"ContainerStarted","Data":"560affbf0fbc2b39b6fc6214aac480cb319f5eb071ad12bbb9f4cd8848293823"}
Oct 30 12:19:44.672186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.672161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs" event={"ID":"0eadd852-c051-4788-a118-9cc91bc2c1ce","Type":"ContainerStarted","Data":"74f2f62bfbce8c2243785b5326ddfb1edc306a5f5aeffa17605f6c1dc50a6981"}
Oct 30 12:19:44.672308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.672207    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs" event={"ID":"0eadd852-c051-4788-a118-9cc91bc2c1ce","Type":"ContainerStarted","Data":"a46ec595cf413a9e2337606f4e608f12bcde2fee21658c95071245d3c2641b92"}
Oct 30 12:19:44.673685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.673665    5060 generic.go:356] "Generic (PLEG): container finished" podID="9781c203-4f34-4803-98a4-ea446780d716" containerID="95001cac3ed4679616415505c5eda485e32364235a8fb515f54dc995688d1905" exitCode=0
Oct 30 12:19:44.673757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.673702    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc" event={"ID":"9781c203-4f34-4803-98a4-ea446780d716","Type":"ContainerDied","Data":"95001cac3ed4679616415505c5eda485e32364235a8fb515f54dc995688d1905"}
Oct 30 12:19:44.715015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.714961    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w" podStartSLOduration=1.714946195 podStartE2EDuration="1.714946195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:44.714290834 +0000 UTC m=+2988.509175328" watchObservedRunningTime="2026-10-30 12:19:44.714946195 +0000 UTC m=+2988.509830690"
Oct 30 12:19:44.799798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:44.799751    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs" podStartSLOduration=1.799737095 podStartE2EDuration="1.799737095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:44.795825142 +0000 UTC m=+2988.590709637" watchObservedRunningTime="2026-10-30 12:19:44.799737095 +0000 UTC m=+2988.594621588"
Oct 30 12:19:45.442293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.442258    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:45.577756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.577724    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:45.684960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.684927    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" containerID="210945bdd03526d5911a09ed503443971fa92b38768b48b54212d484f6fda5e5" exitCode=0
Oct 30 12:19:45.685489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.685014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w" event={"ID":"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52","Type":"ContainerDied","Data":"210945bdd03526d5911a09ed503443971fa92b38768b48b54212d484f6fda5e5"}
Oct 30 12:19:45.687027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.686989    5060 generic.go:356] "Generic (PLEG): container finished" podID="0eadd852-c051-4788-a118-9cc91bc2c1ce" containerID="74f2f62bfbce8c2243785b5326ddfb1edc306a5f5aeffa17605f6c1dc50a6981" exitCode=0
Oct 30 12:19:45.687166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.687062    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs" event={"ID":"0eadd852-c051-4788-a118-9cc91bc2c1ce","Type":"ContainerDied","Data":"74f2f62bfbce8c2243785b5326ddfb1edc306a5f5aeffa17605f6c1dc50a6981"}
Oct 30 12:19:45.890188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:45.890151    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:46.022567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.022482    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca\") pod \"9781c203-4f34-4803-98a4-ea446780d716\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") "
Oct 30 12:19:46.022743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.022677    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m\" (UniqueName: \"kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m\") pod \"9781c203-4f34-4803-98a4-ea446780d716\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") "
Oct 30 12:19:46.022879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.022861    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca\") pod \"9781c203-4f34-4803-98a4-ea446780d716\" (UID: \"9781c203-4f34-4803-98a4-ea446780d716\") "
Oct 30 12:19:46.023173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.023146    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca" pod "9781c203-4f34-4803-98a4-ea446780d716" (UID: "9781c203-4f34-4803-98a4-ea446780d716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:46.023488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.023392    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9781c203-4f34-4803-98a4-ea446780d716-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:46.024903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.024875    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca" pod "9781c203-4f34-4803-98a4-ea446780d716" (UID: "9781c203-4f34-4803-98a4-ea446780d716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:19:46.025314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.025296    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m" pod "9781c203-4f34-4803-98a4-ea446780d716" (UID: "9781c203-4f34-4803-98a4-ea446780d716"). InnerVolumeSpecName "kube-api-access-gj92m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:46.125382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.125350    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9781c203-4f34-4803-98a4-ea446780d716-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:46.125382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.125375    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj92m\" (UniqueName: \"kubernetes.io/projected/9781c203-4f34-4803-98a4-ea446780d716-kube-api-access-gj92m\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:46.450745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.450716    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:19:46.531593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.531355    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"]
Oct 30 12:19:46.699202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.699173    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"
Oct 30 12:19:46.699202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.699188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc" event={"ID":"9781c203-4f34-4803-98a4-ea446780d716","Type":"ContainerDied","Data":"4d4658ca3cc0cced31e4f166e90c8a588dabd1cdfd2e791d4407507b902854be"}
Oct 30 12:19:46.699757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.699246    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4658ca3cc0cced31e4f166e90c8a588dabd1cdfd2e791d4407507b902854be"
Oct 30 12:19:46.931676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.931647    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:46.933949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:46.933926    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:47.039646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.039538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf\" (UniqueName: \"kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf\") pod \"0eadd852-c051-4788-a118-9cc91bc2c1ce\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") "
Oct 30 12:19:47.039646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.039593    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca\") pod \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") "
Oct 30 12:19:47.039908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.039684    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca\") pod \"0eadd852-c051-4788-a118-9cc91bc2c1ce\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") "
Oct 30 12:19:47.039908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.039885    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca\") pod \"0eadd852-c051-4788-a118-9cc91bc2c1ce\" (UID: \"0eadd852-c051-4788-a118-9cc91bc2c1ce\") "
Oct 30 12:19:47.040028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.039914    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx\" (UniqueName: \"kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx\") pod \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") "
Oct 30 12:19:47.040028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.039954    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca\") pod \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\" (UID: \"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52\") "
Oct 30 12:19:47.040964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.040929    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca" pod "0eadd852-c051-4788-a118-9cc91bc2c1ce" (UID: "0eadd852-c051-4788-a118-9cc91bc2c1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:47.041111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.040975    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca" pod "0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" (UID: "0ae53798-5b70-4ad1-a0b1-b8d6995fbb52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:47.042983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.042954    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca" pod "0eadd852-c051-4788-a118-9cc91bc2c1ce" (UID: "0eadd852-c051-4788-a118-9cc91bc2c1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:19:47.043909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.043884    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx" pod "0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" (UID: "0ae53798-5b70-4ad1-a0b1-b8d6995fbb52"). InnerVolumeSpecName "kube-api-access-h86cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:47.044066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.044033    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca" pod "0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" (UID: "0ae53798-5b70-4ad1-a0b1-b8d6995fbb52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:19:47.044198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.044119    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf" pod "0eadd852-c051-4788-a118-9cc91bc2c1ce" (UID: "0eadd852-c051-4788-a118-9cc91bc2c1ce"). InnerVolumeSpecName "kube-api-access-ff6jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:47.142116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.142077    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:47.142116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.142110    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eadd852-c051-4788-a118-9cc91bc2c1ce-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:47.142116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.142120    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86cx\" (UniqueName: \"kubernetes.io/projected/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-kube-api-access-h86cx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:47.142370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.142129    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:47.142370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.142140    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff6jf\" (UniqueName: \"kubernetes.io/projected/0eadd852-c051-4788-a118-9cc91bc2c1ce-kube-api-access-ff6jf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:47.142370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.142148    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:47.350989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.350898    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:47.710646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.710611    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"
Oct 30 12:19:47.711146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.710644    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs" event={"ID":"0eadd852-c051-4788-a118-9cc91bc2c1ce","Type":"ContainerDied","Data":"a46ec595cf413a9e2337606f4e608f12bcde2fee21658c95071245d3c2641b92"}
Oct 30 12:19:47.711146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.710700    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a46ec595cf413a9e2337606f4e608f12bcde2fee21658c95071245d3c2641b92"
Oct 30 12:19:47.712443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.712420    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"
Oct 30 12:19:47.712443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.712418    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w" event={"ID":"0ae53798-5b70-4ad1-a0b1-b8d6995fbb52","Type":"ContainerDied","Data":"560affbf0fbc2b39b6fc6214aac480cb319f5eb071ad12bbb9f4cd8848293823"}
Oct 30 12:19:47.712667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:47.712519    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="560affbf0fbc2b39b6fc6214aac480cb319f5eb071ad12bbb9f4cd8848293823"
Oct 30 12:19:49.104513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:49.104412    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:49.857639 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:49.857601    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:19:50.861832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:50.861797    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:51.531891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.531848    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5" podUID="e51bc15e-9915-4720-be99-f383c4d72f53" containerName="payload-processing" containerID="cri-o://b664ff81dd1c4f4cc3ae517027767003520b14c347b00a42a20745e6648bbfa4" gracePeriod=25
Oct 30 12:19:51.760208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.756764    5060 generic.go:356] "Generic (PLEG): container finished" podID="e51bc15e-9915-4720-be99-f383c4d72f53" containerID="b664ff81dd1c4f4cc3ae517027767003520b14c347b00a42a20745e6648bbfa4" exitCode=0
Oct 30 12:19:51.765309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.763999    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5" event={"ID":"e51bc15e-9915-4720-be99-f383c4d72f53","Type":"ContainerDied","Data":"b664ff81dd1c4f4cc3ae517027767003520b14c347b00a42a20745e6648bbfa4"}
Oct 30 12:19:51.829883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.829858    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:51.907421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.907387    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume\") pod \"e51bc15e-9915-4720-be99-f383c4d72f53\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") "
Oct 30 12:19:51.908918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.907606    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume\") pod \"e51bc15e-9915-4720-be99-f383c4d72f53\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") "
Oct 30 12:19:51.908918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.907708    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials\") pod \"e51bc15e-9915-4720-be99-f383c4d72f53\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") "
Oct 30 12:19:51.908918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.907744    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr\") pod \"e51bc15e-9915-4720-be99-f383c4d72f53\" (UID: \"e51bc15e-9915-4720-be99-f383c4d72f53\") "
Oct 30 12:19:51.908918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.908426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume" pod "e51bc15e-9915-4720-be99-f383c4d72f53" (UID: "e51bc15e-9915-4720-be99-f383c4d72f53"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:51.908918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.908672    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:51.909288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.909157    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume" pod "e51bc15e-9915-4720-be99-f383c4d72f53" (UID: "e51bc15e-9915-4720-be99-f383c4d72f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:51.911654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.911559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr" pod "e51bc15e-9915-4720-be99-f383c4d72f53" (UID: "e51bc15e-9915-4720-be99-f383c4d72f53"). InnerVolumeSpecName "kube-api-access-gxfjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:51.911810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:51.911679    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials" pod "e51bc15e-9915-4720-be99-f383c4d72f53" (UID: "e51bc15e-9915-4720-be99-f383c4d72f53"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:52.009739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.009702    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:52.009739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.009734    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfjr\" (UniqueName: \"kubernetes.io/projected/e51bc15e-9915-4720-be99-f383c4d72f53-kube-api-access-gxfjr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:52.009739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.009744    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51bc15e-9915-4720-be99-f383c4d72f53-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:52.646481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.646375    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:52.774707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.774669    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"
Oct 30 12:19:52.774887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.774705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5" event={"ID":"e51bc15e-9915-4720-be99-f383c4d72f53","Type":"ContainerDied","Data":"90437d486bb326326c44016fb1f782d45c4e3649739c2026a30eb9adf0466d21"}
Oct 30 12:19:52.774887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.774769    5060 scope.go:122] "RemoveContainer" containerID="b664ff81dd1c4f4cc3ae517027767003520b14c347b00a42a20745e6648bbfa4"
Oct 30 12:19:52.829542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.829508    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"]
Oct 30 12:19:52.841287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:52.841256    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97bkpr5"]
Oct 30 12:19:53.092755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.092720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:53.357369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.357279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:53.709349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.709302    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:53.855478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.855097    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:53.995585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.995499    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"]
Oct 30 12:19:53.996279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996257    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0eadd852-c051-4788-a118-9cc91bc2c1ce" containerName="revoke-keys"
Oct 30 12:19:53.996396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996280    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eadd852-c051-4788-a118-9cc91bc2c1ce" containerName="revoke-keys"
Oct 30 12:19:53.996396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996313    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e51bc15e-9915-4720-be99-f383c4d72f53" containerName="payload-processing"
Oct 30 12:19:53.996396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996321    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51bc15e-9915-4720-be99-f383c4d72f53" containerName="payload-processing"
Oct 30 12:19:53.996396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996332    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9781c203-4f34-4803-98a4-ea446780d716" containerName="revoke-keys"
Oct 30 12:19:53.996396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996339    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9781c203-4f34-4803-98a4-ea446780d716" containerName="revoke-keys"
Oct 30 12:19:53.996697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996407    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" containerName="revoke-keys"
Oct 30 12:19:53.996697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996416    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" containerName="revoke-keys"
Oct 30 12:19:53.996697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996615    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" containerName="revoke-keys"
Oct 30 12:19:53.996697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996634    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eadd852-c051-4788-a118-9cc91bc2c1ce" containerName="revoke-keys"
Oct 30 12:19:53.996697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996647    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51bc15e-9915-4720-be99-f383c4d72f53" containerName="payload-processing"
Oct 30 12:19:53.996697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:53.996655    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9781c203-4f34-4803-98a4-ea446780d716" containerName="revoke-keys"
Oct 30 12:19:54.001737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.001716    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.019413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.019388    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"]
Oct 30 12:19:54.028796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.028771    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:19:54.029013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.028771    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5c638a89dd61\""
Oct 30 12:19:54.029104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.028792    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:19:54.029104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.028833    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:19:54.029104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.028855    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:19:54.029447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.029425    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qtlnm\""
Oct 30 12:19:54.098200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.098170    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"]
Oct 30 12:19:54.102711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.102690    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.117670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.117646    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:19:54.117843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.117758    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:19:54.117976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.117954    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:19:54.117976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.117975    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:19:54.122628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.122601    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"]
Oct 30 12:19:54.145342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.145305    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.145503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.145362    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.145581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.145534    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.145788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.145769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb944\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.250322    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.250407    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb944\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.254600    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.254654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.254722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.254878    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.254945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.262526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.260595    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.275475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.270597    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.288483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.285407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb944\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.288483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.285979    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials\") pod \"payload-processing-external-model-66bb89db4d-xrxfp\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") " pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.323477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.323036    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.360196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.357334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.360196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.357406    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.360196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.357475    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.360196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.358357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.370505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.369433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.375831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.375805    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51bc15e-9915-4720-be99-f383c4d72f53" path="/var/lib/kubelet/pods/e51bc15e-9915-4720-be99-f383c4d72f53/volumes"
Oct 30 12:19:54.388438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.388348    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q\") pod \"maas-api-revoke-sub-2fa0d27c-xlfkf\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.424385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.424172    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:54.681828 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:54.681797    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4aa36c1_0379_4e04_a082_f0dcaf1b6cfd.slice/crio-0749a1db9bd1f918b24e2a3b6b92a199cf77f920464fb612229f3b00a40cea60 WatchSource:0}: Error finding container 0749a1db9bd1f918b24e2a3b6b92a199cf77f920464fb612229f3b00a40cea60: Status 404 returned error can't find the container with id 0749a1db9bd1f918b24e2a3b6b92a199cf77f920464fb612229f3b00a40cea60
Oct 30 12:19:54.686128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.686096    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"]
Oct 30 12:19:54.718810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.718778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:54.757678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.757649    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"]
Oct 30 12:19:54.757796 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:19:54.757717    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92844707_ea61_400a_8a3c_e82304b3d122.slice/crio-bf68ace70e6b7a2f7e309a8ac11aa6f0db43614a0c42c717360007d6d3f88b49 WatchSource:0}: Error finding container bf68ace70e6b7a2f7e309a8ac11aa6f0db43614a0c42c717360007d6d3f88b49: Status 404 returned error can't find the container with id bf68ace70e6b7a2f7e309a8ac11aa6f0db43614a0c42c717360007d6d3f88b49
Oct 30 12:19:54.850206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.850168    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" event={"ID":"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd","Type":"ContainerStarted","Data":"5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff"}
Oct 30 12:19:54.850397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.850232    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" event={"ID":"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd","Type":"ContainerStarted","Data":"0749a1db9bd1f918b24e2a3b6b92a199cf77f920464fb612229f3b00a40cea60"}
Oct 30 12:19:54.850397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.850260    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:19:54.863132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.863091    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf" event={"ID":"92844707-ea61-400a-8a3c-e82304b3d122","Type":"ContainerStarted","Data":"bf68ace70e6b7a2f7e309a8ac11aa6f0db43614a0c42c717360007d6d3f88b49"}
Oct 30 12:19:54.895752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:54.895695    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" podStartSLOduration=1.895680806 podStartE2EDuration="1.895680806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:54.89131243 +0000 UTC m=+2998.686196925" watchObservedRunningTime="2026-10-30 12:19:54.895680806 +0000 UTC m=+2998.690565299"
Oct 30 12:19:55.396690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.396523    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"]
Oct 30 12:19:55.433190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.431369    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:55.578251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.578211    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:55.878764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.878719    5060 generic.go:356] "Generic (PLEG): container finished" podID="92844707-ea61-400a-8a3c-e82304b3d122" containerID="353f0db70f13244ffa87299e9ec1e837a2d3320edb06ac8091ec16244a56e858" exitCode=0
Oct 30 12:19:55.879485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.879318    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf" event={"ID":"92844707-ea61-400a-8a3c-e82304b3d122","Type":"ContainerDied","Data":"353f0db70f13244ffa87299e9ec1e837a2d3320edb06ac8091ec16244a56e858"}
Oct 30 12:19:55.880226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.880204    5060 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" secret="" err="secret \"payload-processing-external-model-dockercfg-qtlnm\" not found"
Oct 30 12:19:55.909071 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:55.896983    5060 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5c638a89dd61: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:55.909961 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:55.909939    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:55.910151 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:55.897071    5060 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 30 12:19:55.910606 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:55.910308    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials podName:f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd nodeName:}" failed. No retries permitted until 2026-10-30 12:19:56.410283419 +0000 UTC m=+3000.205167904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials") pod "payload-processing-external-model-66bb89db4d-xrxfp" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd") : secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:55.912032 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:55.912016    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume podName:f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd nodeName:}" failed. No retries permitted until 2026-10-30 12:19:56.411994678 +0000 UTC m=+3000.206879160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume") pod "payload-processing-external-model-66bb89db4d-xrxfp" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd") : configmap "payload-processing-external-model-plugins" not found
Oct 30 12:19:55.957180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:55.957148    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:56.106313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:56.106264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:56.423353 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:56.423268    5060 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5c638a89dd61: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:56.423889 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:56.423870    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:56.424051 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:56.424039    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials podName:f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd nodeName:}" failed. No retries permitted until 2026-10-30 12:19:57.424017698 +0000 UTC m=+3001.218902187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials") pod "payload-processing-external-model-66bb89db4d-xrxfp" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd") : secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:56.425211 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:56.425186    5060 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 30 12:19:56.425407 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:56.425394    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume podName:f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd nodeName:}" failed. No retries permitted until 2026-10-30 12:19:57.425375988 +0000 UTC m=+3001.220260462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume") pod "payload-processing-external-model-66bb89db4d-xrxfp" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd") : configmap "payload-processing-external-model-plugins" not found
Oct 30 12:19:56.792010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:56.791975    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:56.958242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:56.958206    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:57.155133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.155106    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:57.207281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.207230    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:57.355499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.355311    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q\") pod \"92844707-ea61-400a-8a3c-e82304b3d122\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") "
Oct 30 12:19:57.355499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.355475    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca\") pod \"92844707-ea61-400a-8a3c-e82304b3d122\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") "
Oct 30 12:19:57.355780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.355685    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca\") pod \"92844707-ea61-400a-8a3c-e82304b3d122\" (UID: \"92844707-ea61-400a-8a3c-e82304b3d122\") "
Oct 30 12:19:57.382983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.382851    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca" pod "92844707-ea61-400a-8a3c-e82304b3d122" (UID: "92844707-ea61-400a-8a3c-e82304b3d122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:57.396972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.396936    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q" pod "92844707-ea61-400a-8a3c-e82304b3d122" (UID: "92844707-ea61-400a-8a3c-e82304b3d122"). InnerVolumeSpecName "kube-api-access-rrh2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:57.396972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.396962    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca" pod "92844707-ea61-400a-8a3c-e82304b3d122" (UID: "92844707-ea61-400a-8a3c-e82304b3d122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:19:57.399567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.399536    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:57.457891 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:57.457861    5060 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5c638a89dd61: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:57.457891 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:57.457889    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:57.458499 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:57.457940    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials podName:f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd nodeName:}" failed. No retries permitted until 2026-10-30 12:19:59.45792449 +0000 UTC m=+3003.252808963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials") pod "payload-processing-external-model-66bb89db4d-xrxfp" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd") : secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:57.458499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.457863    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:57.458499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.457970    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92844707-ea61-400a-8a3c-e82304b3d122-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:57.458499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.457981    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/92844707-ea61-400a-8a3c-e82304b3d122-kube-api-access-rrh2q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:19:57.529375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.529342    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:19:57.541725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.541696    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:19:57.669613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.669525    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:57.952297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.952269    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"
Oct 30 12:19:57.952488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.952269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf" event={"ID":"92844707-ea61-400a-8a3c-e82304b3d122","Type":"ContainerDied","Data":"bf68ace70e6b7a2f7e309a8ac11aa6f0db43614a0c42c717360007d6d3f88b49"}
Oct 30 12:19:57.952488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:57.952392    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf68ace70e6b7a2f7e309a8ac11aa6f0db43614a0c42c717360007d6d3f88b49"
Oct 30 12:19:58.048184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:58.048143    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"]
Oct 30 12:19:58.051852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:58.051827    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d07102e-4ffvz"]
Oct 30 12:19:58.299990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:58.299898    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:58.336815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:58.336784    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3" path="/var/lib/kubelet/pods/1fd5502c-b8c4-442b-9a0f-4ec1b59c3ec3/volumes"
Oct 30 12:19:58.442443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:58.442390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:59.012481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:19:59.012426    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:19:59.514491 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:59.513376    5060 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5c638a89dd61: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:59.514491 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:59.513399    5060 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp: secret "agc-cred-5c638a89dd61" not found
Oct 30 12:19:59.514491 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:19:59.513463    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials podName:f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd nodeName:}" failed. No retries permitted until 2026-10-30 12:20:03.513432575 +0000 UTC m=+3007.308317046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials") pod "payload-processing-external-model-66bb89db4d-xrxfp" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd") : secret "agc-cred-5c638a89dd61" not found
Oct 30 12:20:00.395723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:00.395692    5060 scope.go:122] "RemoveContainer" containerID="f99410445f791471ee64e53da90d5af1bc4dfe44b2a19b3a164a25da6a24a3d8"
Oct 30 12:20:00.410438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:00.410409    5060 scope.go:122] "RemoveContainer" containerID="0540915c3d763831fdf4a27a1b066ece75133d390eed29ab4c64017e344991af"
Oct 30 12:20:00.530506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:00.530440    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:01.350436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.346139    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:01.494229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.493916    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"]
Oct 30 12:20:01.494772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.494752    5060 cpu_manager.go:414] "containerMap: removing container" podUID="92844707-ea61-400a-8a3c-e82304b3d122" containerName="revoke-keys"
Oct 30 12:20:01.494860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.494774    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92844707-ea61-400a-8a3c-e82304b3d122" containerName="revoke-keys"
Oct 30 12:20:01.494938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.494925    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="92844707-ea61-400a-8a3c-e82304b3d122" containerName="revoke-keys"
Oct 30 12:20:01.552552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.552446    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:01.553098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.553076    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"]
Oct 30 12:20:01.553395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.553331    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.632359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.632266    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5qmrg\""
Oct 30 12:20:01.661940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.661902    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662017    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662153    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662229    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662288    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662353    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662405    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.662597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.662504    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m5c\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764322    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764379    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764418    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764488    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764545    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764596    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764671    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m5c\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764783    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.764926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764826    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.765160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.764998    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.765269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.765248    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.765505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.765479    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.765607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.765510    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.767078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.767016    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.769253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.769181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.853061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.853024    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.855799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.855768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m5c\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fp4jz\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.939515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.939396    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:01.943348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:01.943296    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" podUID="f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" containerName="payload-processing" containerID="cri-o://5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff" gracePeriod=25
Oct 30 12:20:02.231586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.231559    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:20:02.275746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.275708    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials\") pod \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") "
Oct 30 12:20:02.275910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.275776    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944\") pod \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") "
Oct 30 12:20:02.275910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.275852    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume\") pod \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") "
Oct 30 12:20:02.276052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.275921    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume\") pod \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\" (UID: \"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd\") "
Oct 30 12:20:02.277166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.277077    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume" pod "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:02.277166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.277140    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume" pod "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:02.280361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.280334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944" pod "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd"). InnerVolumeSpecName "kube-api-access-xb944". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:02.280922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.280873    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials" pod "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" (UID: "f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:02.282156 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:02.282133    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fdbf34_6a5f_4db3_983e_ba670dc24c27.slice/crio-406a3b790ee2c0a3f490d6467e25e3a13f4727e91af023ebb90076f702be600f WatchSource:0}: Error finding container 406a3b790ee2c0a3f490d6467e25e3a13f4727e91af023ebb90076f702be600f: Status 404 returned error can't find the container with id 406a3b790ee2c0a3f490d6467e25e3a13f4727e91af023ebb90076f702be600f
Oct 30 12:20:02.285808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.285743    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:02.285920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.285858    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:02.285920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.285903    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:02.291744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.291721    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"]
Oct 30 12:20:02.378585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.378549    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:02.378585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.378584    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb944\" (UniqueName: \"kubernetes.io/projected/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-kube-api-access-xb944\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:02.378796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.378611    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:02.378796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.378628    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:02.606599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.606493    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:02.752328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.752292    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:02.904899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:02.904812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:03.097692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.097650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" event={"ID":"64fdbf34-6a5f-4db3-983e-ba670dc24c27","Type":"ContainerStarted","Data":"9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9"}
Oct 30 12:20:03.098003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.097979    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" event={"ID":"64fdbf34-6a5f-4db3-983e-ba670dc24c27","Type":"ContainerStarted","Data":"406a3b790ee2c0a3f490d6467e25e3a13f4727e91af023ebb90076f702be600f"}
Oct 30 12:20:03.104768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.104736    5060 generic.go:356] "Generic (PLEG): container finished" podID="f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" containerID="5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff" exitCode=0
Oct 30 12:20:03.114483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.105093    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"
Oct 30 12:20:03.123477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.105111    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" event={"ID":"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd","Type":"ContainerDied","Data":"5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff"}
Oct 30 12:20:03.124016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.123986    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp" event={"ID":"f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd","Type":"ContainerDied","Data":"0749a1db9bd1f918b24e2a3b6b92a199cf77f920464fb612229f3b00a40cea60"}
Oct 30 12:20:03.124368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.124051    5060 scope.go:122] "RemoveContainer" containerID="5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff"
Oct 30 12:20:03.177730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.177605    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podStartSLOduration=2.177581016 podStartE2EDuration="2.177581016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:03.151439191 +0000 UTC m=+3006.946323688" watchObservedRunningTime="2026-10-30 12:20:03.177581016 +0000 UTC m=+3006.972465512"
Oct 30 12:20:03.212693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.206561    5060 scope.go:122] "RemoveContainer" containerID="5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff"
Oct 30 12:20:03.212693 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:20:03.206978    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff\": container with ID starting with 5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff not found: ID does not exist" containerID="5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff"
Oct 30 12:20:03.212693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.207013    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff"} err="failed to get container status \"5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff\": rpc error: code = NotFound desc = could not find container \"5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff\": container with ID starting with 5bb42588cdc509f4908b225f6f6470868b4e9cfeda071d2c698b0dda9130fcff not found: ID does not exist"
Oct 30 12:20:03.271827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.271790    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"]
Oct 30 12:20:03.280988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.280943    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bb89db4d-xrxfp"]
Oct 30 12:20:03.601480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.596058    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:03.940837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.940752    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:03.942268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.942246    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:03.942359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:03.942297    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:04.336483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:04.336432    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" path="/var/lib/kubelet/pods/f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd/volumes"
Oct 30 12:20:04.940771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:04.940737    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:04.940957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:04.940794    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:05.298477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.297531    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:05.473724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.473184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:05.615232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.615136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:05.800519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.800422    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:05.952535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.941905    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:05.952535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.941987    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:05.970477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:05.967522    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:06.153810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:06.153774    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:06.556434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:06.556337    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:06.714484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:06.710978    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:06.944535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:06.942839    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:06.944535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:06.942899    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:07.168275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.168234    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:07.172301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.172270    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"]
Oct 30 12:20:07.173321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.173299    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" containerName="payload-processing"
Oct 30 12:20:07.173321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.173322    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" containerName="payload-processing"
Oct 30 12:20:07.173605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.173587    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4aa36c1-0379-4e04-a082-f0dcaf1b6cfd" containerName="payload-processing"
Oct 30 12:20:07.364724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.340416    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6452p\" (UniqueName: \"kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.364724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.340602    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.364724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.340675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.383072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.383040    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"]
Oct 30 12:20:07.385626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.384972    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.393510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.388378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:07.435516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.413437    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:20:07.435516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.415475    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:20:07.435516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.415578    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:20:07.435516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.415784    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:20:07.448511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.448482    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6452p\" (UniqueName: \"kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.449176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.449151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.453872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.453849    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.466866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.466838    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.502960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.499921    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.596482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.596332    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6452p\" (UniqueName: \"kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p\") pod \"maas-api-revoke-sub-24dcb68d-vghtx\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.749010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.748979    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:07.942471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.941865    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:07.942471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:07.941925    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:08.130383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:08.129969    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"]
Oct 30 12:20:08.224492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:08.223578    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx" event={"ID":"a6cce878-2a2c-4105-946a-3b78a65531f6","Type":"ContainerStarted","Data":"5c246820d3e5c7e420301e3eeb8bdad94b754264fd32e3db244edcf460175afa"}
Oct 30 12:20:08.255131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:08.255093    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:08.941256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:08.941219    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:08.941432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:08.941277    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:09.234779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:09.234737    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx" event={"ID":"a6cce878-2a2c-4105-946a-3b78a65531f6","Type":"ContainerStarted","Data":"3ce2a97b858d187b2c67eb3db68f7b5392a6843f00d9eff1ff0b9a19fe23d221"}
Oct 30 12:20:09.276661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:09.276608    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx" podStartSLOduration=2.2765943379999998 podStartE2EDuration="2.276594338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:09.274244515 +0000 UTC m=+3013.069129010" watchObservedRunningTime="2026-10-30 12:20:09.276594338 +0000 UTC m=+3013.071478832"
Oct 30 12:20:09.941285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:09.941247    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:09.941495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:09.941300    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:10.076806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.076766    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:10.244475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.242485    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:10.268767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.268731    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6cce878-2a2c-4105-946a-3b78a65531f6" containerID="3ce2a97b858d187b2c67eb3db68f7b5392a6843f00d9eff1ff0b9a19fe23d221" exitCode=0
Oct 30 12:20:10.268939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.268820    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx" event={"ID":"a6cce878-2a2c-4105-946a-3b78a65531f6","Type":"ContainerDied","Data":"3ce2a97b858d187b2c67eb3db68f7b5392a6843f00d9eff1ff0b9a19fe23d221"}
Oct 30 12:20:10.338326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.338292    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"]
Oct 30 12:20:10.344606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.344577    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.360781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.360753    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"]
Oct 30 12:20:10.361780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.361754    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2\""
Oct 30 12:20:10.361984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.361871    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"agc-cred-972432cc9b49\""
Oct 30 12:20:10.361984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.361906    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"openshift-service-ca.crt\""
Oct 30 12:20:10.362176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.362044    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"payload-processing-external-model-e2e--793d1056-dockercfg-zj5q5\""
Oct 30 12:20:10.362176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.362106    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"routing-overlay\""
Oct 30 12:20:10.362176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.362046    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"kube-root-ca.crt\""
Oct 30 12:20:10.460476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.453860    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvkf\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.460476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.454012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.460476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.454132    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.460476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.454316    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.577534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.568363    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.577534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.568526    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.577534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.568631    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvkf\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.577534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.568738    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.577534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.569744    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.577534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.570317    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.578556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.578531    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.609235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.609168    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvkf\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.665496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.665447    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:10.941024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.940941    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 12:20:10.941024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.940999    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 12:20:10.979989 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:10.979941    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bf43af_3e56_497a_bfe5_796c298bfa96.slice/crio-458c66c201d1af20828c0452a0e97168cdb2193b808b6ea8c6d6128cfd81ac00 WatchSource:0}: Error finding container 458c66c201d1af20828c0452a0e97168cdb2193b808b6ea8c6d6128cfd81ac00: Status 404 returned error can't find the container with id 458c66c201d1af20828c0452a0e97168cdb2193b808b6ea8c6d6128cfd81ac00
Oct 30 12:20:10.982720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.982682    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"]
Oct 30 12:20:10.990988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:10.990959    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:11.286077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.285676    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58" event={"ID":"f2bf43af-3e56-497a-bfe5-796c298bfa96","Type":"ContainerStarted","Data":"e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19"}
Oct 30 12:20:11.286077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.285735    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58" event={"ID":"f2bf43af-3e56-497a-bfe5-796c298bfa96","Type":"ContainerStarted","Data":"458c66c201d1af20828c0452a0e97168cdb2193b808b6ea8c6d6128cfd81ac00"}
Oct 30 12:20:11.286077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.285790    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:11.368474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.359687    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58" podStartSLOduration=1.359665182 podStartE2EDuration="1.359665182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:11.33375591 +0000 UTC m=+3015.128640402" watchObservedRunningTime="2026-10-30 12:20:11.359665182 +0000 UTC m=+3015.154549677"
Oct 30 12:20:11.730166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.730134    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:11.831322 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.831228    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca\") pod \"a6cce878-2a2c-4105-946a-3b78a65531f6\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") "
Oct 30 12:20:11.838878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.838832    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca" pod "a6cce878-2a2c-4105-946a-3b78a65531f6" (UID: "a6cce878-2a2c-4105-946a-3b78a65531f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:11.839059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.839046    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p\" (UniqueName: \"kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p\") pod \"a6cce878-2a2c-4105-946a-3b78a65531f6\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") "
Oct 30 12:20:11.839188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.839172    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca\") pod \"a6cce878-2a2c-4105-946a-3b78a65531f6\" (UID: \"a6cce878-2a2c-4105-946a-3b78a65531f6\") "
Oct 30 12:20:11.839740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.839717    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:11.842682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.842652    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p" pod "a6cce878-2a2c-4105-946a-3b78a65531f6" (UID: "a6cce878-2a2c-4105-946a-3b78a65531f6"). InnerVolumeSpecName "kube-api-access-6452p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:11.853690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.853664    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca" pod "a6cce878-2a2c-4105-946a-3b78a65531f6" (UID: "a6cce878-2a2c-4105-946a-3b78a65531f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:20:11.940643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.940606    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6452p\" (UniqueName: \"kubernetes.io/projected/a6cce878-2a2c-4105-946a-3b78a65531f6-kube-api-access-6452p\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:11.940643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:11.940637    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6cce878-2a2c-4105-946a-3b78a65531f6-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:12.083371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:12.083239    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:12.297835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:12.297804    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"
Oct 30 12:20:12.298308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:12.297840    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx" event={"ID":"a6cce878-2a2c-4105-946a-3b78a65531f6","Type":"ContainerDied","Data":"5c246820d3e5c7e420301e3eeb8bdad94b754264fd32e3db244edcf460175afa"}
Oct 30 12:20:12.298308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:12.297888    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c246820d3e5c7e420301e3eeb8bdad94b754264fd32e3db244edcf460175afa"
Oct 30 12:20:12.949958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:12.949910    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:20:12.950402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:12.950299    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:20:13.952172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:13.952071    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:14.003475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:14.002599    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:20:14.003475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:14.002706    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:20:14.127480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:14.114999    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:14.253150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:14.253062    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:15.048605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.048550    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:20:15.049141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.048671    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:20:15.149684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.149647    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:15.150341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.150316    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:15.151684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.151664    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:20:15.454504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.454472    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"]
Oct 30 12:20:15.455178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.455164    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6cce878-2a2c-4105-946a-3b78a65531f6" containerName="revoke-keys"
Oct 30 12:20:15.455228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.455181    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cce878-2a2c-4105-946a-3b78a65531f6" containerName="revoke-keys"
Oct 30 12:20:15.455318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.455309    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cce878-2a2c-4105-946a-3b78a65531f6" containerName="revoke-keys"
Oct 30 12:20:15.459549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.459530    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.477220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.477196    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"]
Oct 30 12:20:15.542188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.542158    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"agc-cred-5ac20fbbf60f\""
Oct 30 12:20:15.581302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.581269    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.581508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.581433    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48r4\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.581508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.581500    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.581730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.581705    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.683498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.683436    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48r4\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.683707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.683513    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.683707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.683662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.683831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.683793    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.684206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.684183    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.684565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.684538    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.686629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.686601    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.707334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.707263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48r4\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.774062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.774033    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:15.949380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:15.949341    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:16.092817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.092789    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"]
Oct 30 12:20:16.093516 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:16.093484    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b9c0d2_88eb_4b96_8dcd_249d153c11c8.slice/crio-29b0f48efec0ab900cce40da9da644d445cd38f37b67da85c700f2f0838a45e0 WatchSource:0}: Error finding container 29b0f48efec0ab900cce40da9da644d445cd38f37b67da85c700f2f0838a45e0: Status 404 returned error can't find the container with id 29b0f48efec0ab900cce40da9da644d445cd38f37b67da85c700f2f0838a45e0
Oct 30 12:20:16.345025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.344185    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw" event={"ID":"39b9c0d2-88eb-4b96-8dcd-249d153c11c8","Type":"ContainerStarted","Data":"1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e"}
Oct 30 12:20:16.345025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.344246    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:16.345025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.344264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw" event={"ID":"39b9c0d2-88eb-4b96-8dcd-249d153c11c8","Type":"ContainerStarted","Data":"29b0f48efec0ab900cce40da9da644d445cd38f37b67da85c700f2f0838a45e0"}
Oct 30 12:20:16.388398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.388329    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw" podStartSLOduration=1.388311737 podStartE2EDuration="1.388311737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:16.384347535 +0000 UTC m=+3020.179232030" watchObservedRunningTime="2026-10-30 12:20:16.388311737 +0000 UTC m=+3020.183196230"
Oct 30 12:20:16.655527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.655435    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"]
Oct 30 12:20:16.664741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.660292    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.699641    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"]
Oct 30 12:20:16.702481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702095    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702187    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702394    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702506    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702626    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsnz\" (UniqueName: \"kubernetes.io/projected/9c55ce47-7db4-4f00-8caa-920ee0869c2a-kube-api-access-kbsnz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702657    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.702844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702743    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.703123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.702939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.804326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804278    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.804563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804346    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.804563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804389    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.804736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804532    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsnz\" (UniqueName: \"kubernetes.io/projected/9c55ce47-7db4-4f00-8caa-920ee0869c2a-kube-api-access-kbsnz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.804821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804775    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.804924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804908    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.804936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805074    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805159    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805384    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805545    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.805744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.805715    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.807733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.807710    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.807879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.807863    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.828263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.828241    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsnz\" (UniqueName: \"kubernetes.io/projected/9c55ce47-7db4-4f00-8caa-920ee0869c2a-kube-api-access-kbsnz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.828263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.828259    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c55ce47-7db4-4f00-8caa-920ee0869c2a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j2htj\" (UID: \"9c55ce47-7db4-4f00-8caa-920ee0869c2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:16.975204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:16.975169    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:17.302189 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:17.302159    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c55ce47_7db4_4f00_8caa_920ee0869c2a.slice/crio-074592095a4d9a767eed448badc234d07897c468bbf818db2ab0e32aeeeff3dd WatchSource:0}: Error finding container 074592095a4d9a767eed448badc234d07897c468bbf818db2ab0e32aeeeff3dd: Status 404 returned error can't find the container with id 074592095a4d9a767eed448badc234d07897c468bbf818db2ab0e32aeeeff3dd
Oct 30 12:20:17.305333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:17.305296    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:17.305478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:17.305369    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:17.305478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:17.305401    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:17.307496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:17.307471    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"]
Oct 30 12:20:17.354407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:17.354380    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" event={"ID":"9c55ce47-7db4-4f00-8caa-920ee0869c2a","Type":"ContainerStarted","Data":"074592095a4d9a767eed448badc234d07897c468bbf818db2ab0e32aeeeff3dd"}
Oct 30 12:20:17.779026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:17.778990    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:18.364496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:18.364441    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" event={"ID":"9c55ce47-7db4-4f00-8caa-920ee0869c2a","Type":"ContainerStarted","Data":"2d72dbed278f95392babbcf613cb8eb8e4482520d90b5122b9aad04c31e7628c"}
Oct 30 12:20:18.458080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:18.458024    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podStartSLOduration=2.458009693 podStartE2EDuration="2.458009693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:18.442158092 +0000 UTC m=+3022.237042607" watchObservedRunningTime="2026-10-30 12:20:18.458009693 +0000 UTC m=+3022.252894262"
Oct 30 12:20:18.975999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:18.975964    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:18.980500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:18.978116    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:18.980500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:18.978177    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:19.535104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.535017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:19.681481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.681019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:19.873171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.873083    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"]
Oct 30 12:20:19.919481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.919041    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.929278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.929242    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"]
Oct 30 12:20:19.936939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.934565    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-4a92de-openshift-default-dockercfg-zjvgn\""
Oct 30 12:20:19.957340 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:20:19.955782    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:20:19.963142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963190    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963280    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtwl\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963365    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963436    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963483    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.963692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.963514    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:19.976045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.976007    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:19.976214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:19.976066    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:20.065292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065252    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065389    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065426    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065504    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065530    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtwl\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065597    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065693    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065772    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.065844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065810    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.066067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.065878    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.066112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.066099    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.066327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.066305    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.068136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.068113    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.068330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.068313    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.106479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.105692    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtwl\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.120521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.120079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.152941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.152822    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:20.244134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.244096    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:20.428115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.426021    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"]
Oct 30 12:20:20.518018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.517951    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:20.615117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.615048    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"]
Oct 30 12:20:20.617686 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:20.617658    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dcc4c7_8e41_4fe5_9bb5_32a5f9f86dcc.slice/crio-56f1690536ee6a14821f2f3f5c417422cdcc589c8a41b1799c93f66150634709 WatchSource:0}: Error finding container 56f1690536ee6a14821f2f3f5c417422cdcc589c8a41b1799c93f66150634709: Status 404 returned error can't find the container with id 56f1690536ee6a14821f2f3f5c417422cdcc589c8a41b1799c93f66150634709
Oct 30 12:20:20.620898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.620866    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:20.620987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.620933    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:20.620987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.620964    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:20.986485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.984942    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:20.986485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:20.985006    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:21.449975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:21.449938    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" event={"ID":"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc","Type":"ContainerStarted","Data":"c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4"}
Oct 30 12:20:21.450166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:21.450023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" event={"ID":"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc","Type":"ContainerStarted","Data":"56f1690536ee6a14821f2f3f5c417422cdcc589c8a41b1799c93f66150634709"}
Oct 30 12:20:21.638756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:21.638698    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:21.976102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:21.976064    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:21.976303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:21.976125    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:22.244386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.244287    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:22.246161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.246128    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:22.246293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.246173    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:22.308678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.308644    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:22.357272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.357216    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podStartSLOduration=3.357200941 podStartE2EDuration="3.357200941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:21.500840974 +0000 UTC m=+3025.295725481" watchObservedRunningTime="2026-10-30 12:20:22.357200941 +0000 UTC m=+3026.152085434"
Oct 30 12:20:22.976186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.976144    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:22.976693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:22.976214    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:23.247495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:23.247363    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:23.247779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:23.247751    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:23.511853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:23.511753    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:23.983480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:23.976502    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:23.983480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:23.976589    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:24.244604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:24.244501    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:24.244604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:24.244580    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:24.976310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:24.976275    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 12:20:24.976601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:24.976340    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 12:20:25.245060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.244954    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:25.245060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.245007    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:25.302476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.294096    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:25.426747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.426697    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4" podUID="f173f1ce-9016-43f6-8618-f1a2f94f74e9" containerName="payload-processing" containerID="cri-o://586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70" gracePeriod=25
Oct 30 12:20:25.810616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.810164    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:20:25.925041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.924999    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume\") pod \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") "
Oct 30 12:20:25.925592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.925563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume\") pod \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") "
Oct 30 12:20:25.925928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.925905    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x\") pod \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") "
Oct 30 12:20:25.926111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.926093    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials\") pod \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\" (UID: \"f173f1ce-9016-43f6-8618-f1a2f94f74e9\") "
Oct 30 12:20:25.927730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.927707    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume" pod "f173f1ce-9016-43f6-8618-f1a2f94f74e9" (UID: "f173f1ce-9016-43f6-8618-f1a2f94f74e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:25.929502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.929428    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials" pod "f173f1ce-9016-43f6-8618-f1a2f94f74e9" (UID: "f173f1ce-9016-43f6-8618-f1a2f94f74e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:25.932494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.931857    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume" pod "f173f1ce-9016-43f6-8618-f1a2f94f74e9" (UID: "f173f1ce-9016-43f6-8618-f1a2f94f74e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:25.935734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:25.935712    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x" pod "f173f1ce-9016-43f6-8618-f1a2f94f74e9" (UID: "f173f1ce-9016-43f6-8618-f1a2f94f74e9"). InnerVolumeSpecName "kube-api-access-8pc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:26.029279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.029238    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:26.029279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.029281    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pc2x\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-kube-api-access-8pc2x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:26.029554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.029297    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f173f1ce-9016-43f6-8618-f1a2f94f74e9-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:26.029554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.029314    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173f1ce-9016-43f6-8618-f1a2f94f74e9-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:26.248480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.248010    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:26.248480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.248086    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:26.825881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.825835    5060 generic.go:356] "Generic (PLEG): container finished" podID="f173f1ce-9016-43f6-8618-f1a2f94f74e9" containerID="586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70" exitCode=0
Oct 30 12:20:26.826296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.826264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4" event={"ID":"f173f1ce-9016-43f6-8618-f1a2f94f74e9","Type":"ContainerDied","Data":"586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70"}
Oct 30 12:20:26.826548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.826524    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4" event={"ID":"f173f1ce-9016-43f6-8618-f1a2f94f74e9","Type":"ContainerDied","Data":"fc0e26346b56ee7d7ebebac960b17148d2d1cd25022963452d51f1cb6804753e"}
Oct 30 12:20:26.826700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.826684    5060 scope.go:122] "RemoveContainer" containerID="586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70"
Oct 30 12:20:26.827016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.826999    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"
Oct 30 12:20:26.949918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:26.949882    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"]
Oct 30 12:20:27.014066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.014027    5060 scope.go:122] "RemoveContainer" containerID="586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70"
Oct 30 12:20:27.014894 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:20:27.014866    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70\": container with ID starting with 586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70 not found: ID does not exist" containerID="586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70"
Oct 30 12:20:27.015095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.015067    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70"} err="failed to get container status \"586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70\": rpc error: code = NotFound desc = could not find container \"586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70\": container with ID starting with 586cf5277d7df4e615ee1d3c1840e8eb55e34c14cea68457b5eefc23c71e1c70 not found: ID does not exist"
Oct 30 12:20:27.015400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.015380    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-5f9fs2tz4"]
Oct 30 12:20:27.072102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.072063    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:20:27.089183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.089095    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 12:20:27.090026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.089400    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:20:27.181500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.171223    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:27.274116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.274078    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:27.274746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.274712    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:27.327564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.327378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:27.401536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.399639    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:20:27.567421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.548577    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"]
Oct 30 12:20:27.757614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:27.757502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:28.099659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.099554    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j2htj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:20:28.099990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.099957    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj" podUID="9c55ce47-7db4-4f00-8caa-920ee0869c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:20:28.258481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.247130    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Oct 30 12:20:28.258481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.247188    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Oct 30 12:20:28.271786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.270779    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:28.271786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.271672    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:28.274551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.273054    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j2htj"
Oct 30 12:20:28.358731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:28.358647    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f173f1ce-9016-43f6-8618-f1a2f94f74e9" path="/var/lib/kubelet/pods/f173f1ce-9016-43f6-8618-f1a2f94f74e9/volumes"
Oct 30 12:20:29.251686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:29.251643    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:29.252291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:29.252258    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:29.253169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:29.253147    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:20:29.706617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:29.706582    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:30.036083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:30.035990    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"]
Oct 30 12:20:30.043666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:30.043635    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7adcf4c-rpjff"]
Oct 30 12:20:30.339051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:30.338960    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400d405b-a148-4d3d-a0a8-a7c294372c52" path="/var/lib/kubelet/pods/400d405b-a148-4d3d-a0a8-a7c294372c52/volumes"
Oct 30 12:20:31.547982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:31.547947    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:32.568410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.568361    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58" podUID="f2bf43af-3e56-497a-bfe5-796c298bfa96" containerName="payload-processing" containerID="cri-o://e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19" gracePeriod=25
Oct 30 12:20:32.871623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.871595    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:32.928999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.928951    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials\") pod \"f2bf43af-3e56-497a-bfe5-796c298bfa96\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") "
Oct 30 12:20:32.929199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.929081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume\") pod \"f2bf43af-3e56-497a-bfe5-796c298bfa96\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") "
Oct 30 12:20:32.929199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.929193    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf\") pod \"f2bf43af-3e56-497a-bfe5-796c298bfa96\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") "
Oct 30 12:20:32.929347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.929245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume\") pod \"f2bf43af-3e56-497a-bfe5-796c298bfa96\" (UID: \"f2bf43af-3e56-497a-bfe5-796c298bfa96\") "
Oct 30 12:20:32.929685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.929650    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume" pod "f2bf43af-3e56-497a-bfe5-796c298bfa96" (UID: "f2bf43af-3e56-497a-bfe5-796c298bfa96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:32.929827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.929698    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume" pod "f2bf43af-3e56-497a-bfe5-796c298bfa96" (UID: "f2bf43af-3e56-497a-bfe5-796c298bfa96"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:32.930096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.930071    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:32.930188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.930101    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2bf43af-3e56-497a-bfe5-796c298bfa96-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:32.931938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.931899    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials" pod "f2bf43af-3e56-497a-bfe5-796c298bfa96" (UID: "f2bf43af-3e56-497a-bfe5-796c298bfa96"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:32.933787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.933761    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf" pod "f2bf43af-3e56-497a-bfe5-796c298bfa96" (UID: "f2bf43af-3e56-497a-bfe5-796c298bfa96"). InnerVolumeSpecName "kube-api-access-smvkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:32.972931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.972892    5060 generic.go:356] "Generic (PLEG): container finished" podID="f2bf43af-3e56-497a-bfe5-796c298bfa96" containerID="e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19" exitCode=0
Oct 30 12:20:32.973072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.972945    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"
Oct 30 12:20:32.973072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.972979    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58" event={"ID":"f2bf43af-3e56-497a-bfe5-796c298bfa96","Type":"ContainerDied","Data":"e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19"}
Oct 30 12:20:32.973072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.973036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58" event={"ID":"f2bf43af-3e56-497a-bfe5-796c298bfa96","Type":"ContainerDied","Data":"458c66c201d1af20828c0452a0e97168cdb2193b808b6ea8c6d6128cfd81ac00"}
Oct 30 12:20:32.973072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.973062    5060 scope.go:122] "RemoveContainer" containerID="e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19"
Oct 30 12:20:32.986854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.986825    5060 scope.go:122] "RemoveContainer" containerID="e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19"
Oct 30 12:20:32.987262 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:20:32.987224    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19\": container with ID starting with e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19 not found: ID does not exist" containerID="e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19"
Oct 30 12:20:32.987418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:32.987267    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19"} err="failed to get container status \"e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19\": rpc error: code = NotFound desc = could not find container \"e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19\": container with ID starting with e23966d69199d6d269c896d4219adef353e146fa733a8bab5b71a77f668e7c19 not found: ID does not exist"
Oct 30 12:20:33.031085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.031045    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:33.031085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.031072    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smvkf\" (UniqueName: \"kubernetes.io/projected/f2bf43af-3e56-497a-bfe5-796c298bfa96-kube-api-access-smvkf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:33.032655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.032636    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"]
Oct 30 12:20:33.036934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.036910    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94cms58"]
Oct 30 12:20:33.330196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.330146    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:33.469871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.469841    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:33.889326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:33.889281    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:34.297760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.297714    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"]
Oct 30 12:20:34.298651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.298624    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2bf43af-3e56-497a-bfe5-796c298bfa96" containerName="payload-processing"
Oct 30 12:20:34.298651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.298649    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2bf43af-3e56-497a-bfe5-796c298bfa96" containerName="payload-processing"
Oct 30 12:20:34.298872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.298681    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f173f1ce-9016-43f6-8618-f1a2f94f74e9" containerName="payload-processing"
Oct 30 12:20:34.298872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.298689    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f173f1ce-9016-43f6-8618-f1a2f94f74e9" containerName="payload-processing"
Oct 30 12:20:34.298980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.298892    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2bf43af-3e56-497a-bfe5-796c298bfa96" containerName="payload-processing"
Oct 30 12:20:34.298980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.298910    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f173f1ce-9016-43f6-8618-f1a2f94f74e9" containerName="payload-processing"
Oct 30 12:20:34.304760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.304733    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.335497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.334581    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a1f322-openshift-default-dockercfg-hxsqp\""
Oct 30 12:20:34.344602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.344568    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2bf43af-3e56-497a-bfe5-796c298bfa96" path="/var/lib/kubelet/pods/f2bf43af-3e56-497a-bfe5-796c298bfa96/volumes"
Oct 30 12:20:34.344984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.344968    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"]
Oct 30 12:20:34.351317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.351280    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8bj\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.351683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.351655    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.352074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.351928    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.352290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.352270    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.352513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.352493    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.352733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.352715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.352978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.352959    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.353232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.353206    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.353349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.353257    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.382122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.382087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:34.456477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.456376    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.456727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.456521    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.456727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.456633    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.469286    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.469986    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.470467    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.470550    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8bj\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.470659    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.470746    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.470957    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.471018    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.471910    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.472160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.472611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.472401    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.502341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.495040    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.502341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.501772    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.507216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.507187    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.539942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.539906    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:34.546471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.543778    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8bj\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:34.643890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:34.643784    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:35.030761 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:35.030723    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98605626_4680_4b6c_a7fc_59b59c37d854.slice/crio-7705b5bcc08b1ffba0aab5bb56221e9945a46ac919821d9cb418452e1660fd40 WatchSource:0}: Error finding container 7705b5bcc08b1ffba0aab5bb56221e9945a46ac919821d9cb418452e1660fd40: Status 404 returned error can't find the container with id 7705b5bcc08b1ffba0aab5bb56221e9945a46ac919821d9cb418452e1660fd40
Oct 30 12:20:35.030761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:35.030729    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"]
Oct 30 12:20:35.034586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:35.034552    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:35.034705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:35.034636    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:35.034705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:35.034680    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:20:35.266479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:35.265368    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:36.005970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:36.005924    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" event={"ID":"98605626-4680-4b6c-a7fc-59b59c37d854","Type":"ContainerStarted","Data":"83c3140f31d0abdce33683c9ba67b12b2becc7336770d3129f7beb8340cce8a3"}
Oct 30 12:20:36.006197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:36.005996    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" event={"ID":"98605626-4680-4b6c-a7fc-59b59c37d854","Type":"ContainerStarted","Data":"7705b5bcc08b1ffba0aab5bb56221e9945a46ac919821d9cb418452e1660fd40"}
Oct 30 12:20:36.064418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:36.064365    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podStartSLOduration=2.064351798 podStartE2EDuration="2.064351798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:36.06347227 +0000 UTC m=+3039.858356758" watchObservedRunningTime="2026-10-30 12:20:36.064351798 +0000 UTC m=+3039.859236294"
Oct 30 12:20:36.645288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:36.645254    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:36.651609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:36.651585    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:37.016978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:37.016944    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:37.018030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:37.018003    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:20:37.052756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:37.052722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:38.611640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:38.611606    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:38.757683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:38.757647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:38.902901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:38.902794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:40.597608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:40.597566    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:40.740778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:40.740706    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:40.903174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:40.903051    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:41.000903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.000862    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"]
Oct 30 12:20:41.032007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.031977    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.033293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.033160    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"]
Oct 30 12:20:41.050897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.050736    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:20:41.050897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.050860    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:20:41.062370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.062342    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:20:41.063687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.063661    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:20:41.129056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.129010    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.129270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.129066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.129270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.129154    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtz5\" (UniqueName: \"kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.230949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.230910    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.231177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.230973    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.231177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.231004    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtz5\" (UniqueName: \"kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.231718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.231695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.233664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.233637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.254809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.254766    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtz5\" (UniqueName: \"kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5\") pod \"maas-api-revoke-sub-a2a9228d-l2fr9\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.371628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.371595    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:41.682949 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:20:41.682921    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a51b58_6441_4658_b967_be80a9d581da.slice/crio-a757efcee2d7353ef77cb55fe82c2421c81552f1f8feb9f39dac24b3616ed405 WatchSource:0}: Error finding container a757efcee2d7353ef77cb55fe82c2421c81552f1f8feb9f39dac24b3616ed405: Status 404 returned error can't find the container with id a757efcee2d7353ef77cb55fe82c2421c81552f1f8feb9f39dac24b3616ed405
Oct 30 12:20:41.685498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:41.685436    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"]
Oct 30 12:20:42.075348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:42.075307    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9" event={"ID":"e3a51b58-6441-4658-b967-be80a9d581da","Type":"ContainerStarted","Data":"a757efcee2d7353ef77cb55fe82c2421c81552f1f8feb9f39dac24b3616ed405"}
Oct 30 12:20:42.657350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:42.657315    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:43.087982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:43.087945    5060 generic.go:356] "Generic (PLEG): container finished" podID="e3a51b58-6441-4658-b967-be80a9d581da" containerID="95ff4363c009b81f4f6eec0f43fee93110ec3dee6cdf0a5a7ca9894d59fa8277" exitCode=0
Oct 30 12:20:43.088441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:43.088032    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9" event={"ID":"e3a51b58-6441-4658-b967-be80a9d581da","Type":"ContainerDied","Data":"95ff4363c009b81f4f6eec0f43fee93110ec3dee6cdf0a5a7ca9894d59fa8277"}
Oct 30 12:20:43.100882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:43.100855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:43.241488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:43.241437    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:44.284994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.284961    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:44.347532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.347492    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:44.383434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.383338    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca\") pod \"e3a51b58-6441-4658-b967-be80a9d581da\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") "
Oct 30 12:20:44.383906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.383881    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca\") pod \"e3a51b58-6441-4658-b967-be80a9d581da\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") "
Oct 30 12:20:44.384211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.384186    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5\" (UniqueName: \"kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5\") pod \"e3a51b58-6441-4658-b967-be80a9d581da\" (UID: \"e3a51b58-6441-4658-b967-be80a9d581da\") "
Oct 30 12:20:44.384793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.383930    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca" pod "e3a51b58-6441-4658-b967-be80a9d581da" (UID: "e3a51b58-6441-4658-b967-be80a9d581da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:44.385937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.385910    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:44.400965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.400668    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca" pod "e3a51b58-6441-4658-b967-be80a9d581da" (UID: "e3a51b58-6441-4658-b967-be80a9d581da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:20:44.400965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.400721    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5" pod "e3a51b58-6441-4658-b967-be80a9d581da" (UID: "e3a51b58-6441-4658-b967-be80a9d581da"). InnerVolumeSpecName "kube-api-access-gvtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:44.488350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.488315    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a51b58-6441-4658-b967-be80a9d581da-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:44.488350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:44.488352    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvtz5\" (UniqueName: \"kubernetes.io/projected/e3a51b58-6441-4658-b967-be80a9d581da-kube-api-access-gvtz5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:45.107421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:45.107386    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"
Oct 30 12:20:45.107626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:45.107419    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9" event={"ID":"e3a51b58-6441-4658-b967-be80a9d581da","Type":"ContainerDied","Data":"a757efcee2d7353ef77cb55fe82c2421c81552f1f8feb9f39dac24b3616ed405"}
Oct 30 12:20:45.107626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:45.107488    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a757efcee2d7353ef77cb55fe82c2421c81552f1f8feb9f39dac24b3616ed405"
Oct 30 12:20:46.055115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:46.055082    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:47.864699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:47.864649    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:49.553886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:49.553846    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:49.857959 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:20:49.857878    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:20:51.299307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:51.299270    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:52.336430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:52.336396    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"]
Oct 30 12:20:52.442028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:52.441984    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:52.586248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:52.586207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:52.801478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:52.791284    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:53.287362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:53.287316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:53.446539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:53.445356    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:53.677703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:53.677606    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:54.049894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:54.049852    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:54.222479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:54.220192    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:55.451674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:55.451638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:57.254489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.254415    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:57.337336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.337294    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl" podUID="7f9f1251-c3d0-4545-993d-ba423a6e1264" containerName="payload-processing" containerID="cri-o://1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32" gracePeriod=25
Oct 30 12:20:57.641662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.641633    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:20:57.743203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.743162    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g\") pod \"7f9f1251-c3d0-4545-993d-ba423a6e1264\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") "
Oct 30 12:20:57.743382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.743247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials\") pod \"7f9f1251-c3d0-4545-993d-ba423a6e1264\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") "
Oct 30 12:20:57.743382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.743274    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume\") pod \"7f9f1251-c3d0-4545-993d-ba423a6e1264\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") "
Oct 30 12:20:57.743532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.743506    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume\") pod \"7f9f1251-c3d0-4545-993d-ba423a6e1264\" (UID: \"7f9f1251-c3d0-4545-993d-ba423a6e1264\") "
Oct 30 12:20:57.743752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.743731    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume" pod "7f9f1251-c3d0-4545-993d-ba423a6e1264" (UID: "7f9f1251-c3d0-4545-993d-ba423a6e1264"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:57.743867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.743844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume" pod "7f9f1251-c3d0-4545-993d-ba423a6e1264" (UID: "7f9f1251-c3d0-4545-993d-ba423a6e1264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:57.744161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.744126    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:57.744161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.744145    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9f1251-c3d0-4545-993d-ba423a6e1264-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:57.746139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.746104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g" pod "7f9f1251-c3d0-4545-993d-ba423a6e1264" (UID: "7f9f1251-c3d0-4545-993d-ba423a6e1264"). InnerVolumeSpecName "kube-api-access-v6c9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:57.746139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.746114    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials" pod "7f9f1251-c3d0-4545-993d-ba423a6e1264" (UID: "7f9f1251-c3d0-4545-993d-ba423a6e1264"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:57.756512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.756479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:57.845886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.845796    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6c9g\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-kube-api-access-v6c9g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:57.845886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.845829    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f9f1251-c3d0-4545-993d-ba423a6e1264-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:20:57.915142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:57.915105    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:58.257789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.257751    5060 generic.go:356] "Generic (PLEG): container finished" podID="7f9f1251-c3d0-4545-993d-ba423a6e1264" containerID="1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32" exitCode=0
Oct 30 12:20:58.258321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.257808    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"
Oct 30 12:20:58.258321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.257836    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl" event={"ID":"7f9f1251-c3d0-4545-993d-ba423a6e1264","Type":"ContainerDied","Data":"1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32"}
Oct 30 12:20:58.258321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.257894    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl" event={"ID":"7f9f1251-c3d0-4545-993d-ba423a6e1264","Type":"ContainerDied","Data":"12eb1458a90099177d3343afbe6cbf6a236d88615662f5aef5855037b24b74a4"}
Oct 30 12:20:58.258321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.257923    5060 scope.go:122] "RemoveContainer" containerID="1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32"
Oct 30 12:20:58.271095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.271071    5060 scope.go:122] "RemoveContainer" containerID="1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32"
Oct 30 12:20:58.271446 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:20:58.271426    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32\": container with ID starting with 1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32 not found: ID does not exist" containerID="1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32"
Oct 30 12:20:58.271519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.271501    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32"} err="failed to get container status \"1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32\": rpc error: code = NotFound desc = could not find container \"1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32\": container with ID starting with 1a6a5b986a623e4375f641bcf5b3b305c6d66d0ea6de7929cd44e5d214ee8c32 not found: ID does not exist"
Oct 30 12:20:58.319934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.319893    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"]
Oct 30 12:20:58.337548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:58.337510    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-67e819/payload-processing-external-model-e2e-extmod-a-67e819-6dd4xt9xl"]
Oct 30 12:20:59.155680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:59.155638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:59.620083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:59.620041    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:20:59.832387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:20:59.832135    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:00.202190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.202148    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"]
Oct 30 12:21:00.204327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.204245    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e3a51b58-6441-4658-b967-be80a9d581da" containerName="revoke-keys"
Oct 30 12:21:00.210526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.210491    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a51b58-6441-4658-b967-be80a9d581da" containerName="revoke-keys"
Oct 30 12:21:00.210852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.210837    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7f9f1251-c3d0-4545-993d-ba423a6e1264" containerName="payload-processing"
Oct 30 12:21:00.210957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.210946    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9f1251-c3d0-4545-993d-ba423a6e1264" containerName="payload-processing"
Oct 30 12:21:00.211478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.211433    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9f1251-c3d0-4545-993d-ba423a6e1264" containerName="payload-processing"
Oct 30 12:21:00.213551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.213530    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a51b58-6441-4658-b967-be80a9d581da" containerName="revoke-keys"
Oct 30 12:21:00.230019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.229987    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.231784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.231333    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"]
Oct 30 12:21:00.260170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.260046    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5cf206\""
Oct 30 12:21:00.273569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.273240    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5cf206-dockercfg-prp7k\""
Oct 30 12:21:00.296691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.296638    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.297303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.297265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpt8f\" (UniqueName: \"kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f\") pod \"payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.338240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.338202    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9f1251-c3d0-4545-993d-ba423a6e1264" path="/var/lib/kubelet/pods/7f9f1251-c3d0-4545-993d-ba423a6e1264/volumes"
Oct 30 12:21:00.400252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.400203    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.400438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.400370    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpt8f\" (UniqueName: \"kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f\") pod \"payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.401124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.401096    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.424376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.424340    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpt8f\" (UniqueName: \"kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f\") pod \"payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.491803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.491713    5060 scope.go:122] "RemoveContainer" containerID="d981fa8b05883377e21ae253648b1b5d9e00c56c39c2da8d1dccf6f1cd8fbdbe"
Oct 30 12:21:00.503760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.503737    5060 scope.go:122] "RemoveContainer" containerID="0c30e0a9654879bb27dec5b77005b531a36edf99a4209e59c04462d074aa8806"
Oct 30 12:21:00.576940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.576899    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:00.888467 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:21:00.888407    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ac4633_3d9a_4d1c_aa26_744a86463c46.slice/crio-ebe008eec5ba97319361cba897266e20b0356ab62e05ca9231a2715a6f53427a WatchSource:0}: Error finding container ebe008eec5ba97319361cba897266e20b0356ab62e05ca9231a2715a6f53427a: Status 404 returned error can't find the container with id ebe008eec5ba97319361cba897266e20b0356ab62e05ca9231a2715a6f53427a
Oct 30 12:21:00.892704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:00.892676    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"]
Oct 30 12:21:01.150472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.150364    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:01.308339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.308301    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z" event={"ID":"29ac4633-3d9a-4d1c-aa26-744a86463c46","Type":"ContainerStarted","Data":"071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53"}
Oct 30 12:21:01.308620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.308363    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z" event={"ID":"29ac4633-3d9a-4d1c-aa26-744a86463c46","Type":"ContainerStarted","Data":"ebe008eec5ba97319361cba897266e20b0356ab62e05ca9231a2715a6f53427a"}
Oct 30 12:21:01.308620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.308395    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:01.351212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.351152    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z" podStartSLOduration=2.351132978 podStartE2EDuration="2.351132978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:01.350446543 +0000 UTC m=+3065.145331059" watchObservedRunningTime="2026-10-30 12:21:01.351132978 +0000 UTC m=+3065.146017473"
Oct 30 12:21:01.687499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.687463    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"]
Oct 30 12:21:01.692217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.692191    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.727700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.727663    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"]
Oct 30 12:21:01.731507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.731447    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.731706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.731581    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.731820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.731800    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.731885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.731856    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hn6k\" (UniqueName: \"kubernetes.io/projected/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-kube-api-access-7hn6k\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.732020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.731998    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.732076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.732048    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.732165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.732144    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.732380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.732360    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.732447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.732428    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834244    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834510    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834564    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834597    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.834949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.834823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.835188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.835151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hn6k\" (UniqueName: \"kubernetes.io/projected/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-kube-api-access-7hn6k\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.835188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.835050    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.835393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.835088    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.835393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.835256    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.835708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.835674    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.835845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.835721    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.837609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.837579    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.837919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.837898    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.859258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.859228    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:01.859486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:01.859442    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hn6k\" (UniqueName: \"kubernetes.io/projected/f26c00f4-7dcf-42fc-bf26-bbaddc9ea873-kube-api-access-7hn6k\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mnfsj\" (UID: \"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:02.009107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:02.009068    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:02.335724 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:21:02.335691    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26c00f4_7dcf_42fc_bf26_bbaddc9ea873.slice/crio-eb7581857db6c3df822a21258ed942b445d125a73cd23d2f04a400ceba97d5c1 WatchSource:0}: Error finding container eb7581857db6c3df822a21258ed942b445d125a73cd23d2f04a400ceba97d5c1: Status 404 returned error can't find the container with id eb7581857db6c3df822a21258ed942b445d125a73cd23d2f04a400ceba97d5c1
Oct 30 12:21:02.338704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:02.338670    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:21:02.338800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:02.338739    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:21:02.338800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:02.338770    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:21:02.340307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:02.340280    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"]
Oct 30 12:21:02.856533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:02.856499    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:03.328356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:03.328318    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" event={"ID":"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873","Type":"ContainerStarted","Data":"9760b113e1f35ef67e46b51ea33fd34b19a5d66b1974950fd2d161d747355d5e"}
Oct 30 12:21:03.328867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:03.328379    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" event={"ID":"f26c00f4-7dcf-42fc-bf26-bbaddc9ea873","Type":"ContainerStarted","Data":"eb7581857db6c3df822a21258ed942b445d125a73cd23d2f04a400ceba97d5c1"}
Oct 30 12:21:03.407147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:03.407086    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podStartSLOduration=2.40706879 podStartE2EDuration="2.40706879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:03.388929047 +0000 UTC m=+3067.183813580" watchObservedRunningTime="2026-10-30 12:21:03.40706879 +0000 UTC m=+3067.201953283"
Oct 30 12:21:04.009629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:04.009572    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:04.011729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:04.011697    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:04.011893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:04.011752    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:04.617923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:04.617878    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:05.009372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:05.009335    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:05.009631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:05.009400    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:06.010043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:06.010000    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:06.010629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:06.010058    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:06.448573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:06.448513    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:07.009901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:07.009868    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:07.010199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:07.010156    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:08.010176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:08.010139    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:08.010387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:08.010200    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:08.358382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:08.358284    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:09.009625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:09.009579    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:09.009825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:09.009674    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:10.010428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:10.010389    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 30 12:21:10.010949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:10.010499    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 30 12:21:10.143250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:10.143185    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:11.015063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:11.015029    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:11.015618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:11.015591    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:11.016580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:11.016555    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj"
Oct 30 12:21:11.965551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:11.963479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:12.321332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:12.321231    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:21:13.743000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:13.742964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:15.574878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:15.574837    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:17.258401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:17.258360    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:19.022688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:19.022637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:19.858592 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:21:19.858557    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:21:20.760754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:20.760720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:22.522363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:22.522322    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:22.677283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:22.677232    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:24.557823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:24.557783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:26.362370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:26.362332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:28.123349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:28.123308    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:29.955752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:29.955720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:31.623762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:31.623721    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:33.344182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:33.344142    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:35.080258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:35.080216    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:36.831954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:36.831862    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:38.558873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:38.558838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:40.331835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:40.331800    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:41.992780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:41.992739    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:42.162736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:42.162694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:42.306673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:42.306575    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:43.845659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:43.845610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:45.581935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:45.581894    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:47.344123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:47.344083    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:49.071910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:49.071876    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:49.858365 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:21:49.858317    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:21:50.835192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:50.835156    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:52.711287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:52.711231    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:53.432478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:53.416296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:54.664119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:54.664085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:56.550078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:56.550042    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:58.263428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:58.263373    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:21:59.989743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:21:59.989702    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:00.045246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.045208    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"]
Oct 30 12:22:00.050598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.050572    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1a53f3-nfx7d"]
Oct 30 12:22:00.115579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.115540    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"]
Oct 30 12:22:00.120354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.120326    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.136009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.135974    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:22:00.136198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.136110    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:22:00.136198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.136173    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:22:00.136607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.136582    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:22:00.140600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.140569    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"]
Oct 30 12:22:00.220662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.220605    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.220881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.220776    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.220938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.220907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gr6\" (UniqueName: \"kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.322494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.322377    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gr6\" (UniqueName: \"kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.322684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.322564    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.322684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.322645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.323248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.323212    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.325387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.325353    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.339525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.339444    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10aa4d9-934a-4c7c-8e54-bec3a6b6407c" path="/var/lib/kubelet/pods/b10aa4d9-934a-4c7c-8e54-bec3a6b6407c/volumes"
Oct 30 12:22:00.346616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.346593    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gr6\" (UniqueName: \"kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6\") pod \"maas-api-revoke-sub-fefeb1eb-vkv87\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.436268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.436228    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:00.574828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.574731    5060 scope.go:122] "RemoveContainer" containerID="285bcfa4c0eb73548b1dd43e4691277a4907f135252fdee86b7570f3bd6c4580"
Oct 30 12:22:00.745256 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:22:00.745224    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cda18ac_ae67_4bbd_be13_e386dbf09a34.slice/crio-1bb222518852f5c7658e01183354692d45c4f63dffd5ff31081adead8a84f135 WatchSource:0}: Error finding container 1bb222518852f5c7658e01183354692d45c4f63dffd5ff31081adead8a84f135: Status 404 returned error can't find the container with id 1bb222518852f5c7658e01183354692d45c4f63dffd5ff31081adead8a84f135
Oct 30 12:22:00.746255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.746230    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"]
Oct 30 12:22:00.923043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:00.922949    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87" event={"ID":"4cda18ac-ae67-4bbd-be13-e386dbf09a34","Type":"ContainerStarted","Data":"1bb222518852f5c7658e01183354692d45c4f63dffd5ff31081adead8a84f135"}
Oct 30 12:22:01.682900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:01.682187    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:01.934272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:01.934188    5060 generic.go:356] "Generic (PLEG): container finished" podID="4cda18ac-ae67-4bbd-be13-e386dbf09a34" containerID="9e28012345e33d3ef7481316a5b622ae41439417e3a735a5802ff55c616f3a6e" exitCode=0
Oct 30 12:22:01.934443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:01.934270    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87" event={"ID":"4cda18ac-ae67-4bbd-be13-e386dbf09a34","Type":"ContainerDied","Data":"9e28012345e33d3ef7481316a5b622ae41439417e3a735a5802ff55c616f3a6e"}
Oct 30 12:22:03.124573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.124544    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:03.166232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.166203    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca\") pod \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") "
Oct 30 12:22:03.166445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.166413    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6\" (UniqueName: \"kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6\") pod \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") "
Oct 30 12:22:03.166596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.166512    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca\") pod \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\" (UID: \"4cda18ac-ae67-4bbd-be13-e386dbf09a34\") "
Oct 30 12:22:03.169162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.169124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca" pod "4cda18ac-ae67-4bbd-be13-e386dbf09a34" (UID: "4cda18ac-ae67-4bbd-be13-e386dbf09a34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:22:03.169718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.169692    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:03.170999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.170968    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6" pod "4cda18ac-ae67-4bbd-be13-e386dbf09a34" (UID: "4cda18ac-ae67-4bbd-be13-e386dbf09a34"). InnerVolumeSpecName "kube-api-access-k8gr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:03.189480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.189415    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca" pod "4cda18ac-ae67-4bbd-be13-e386dbf09a34" (UID: "4cda18ac-ae67-4bbd-be13-e386dbf09a34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:03.271079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.271040    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8gr6\" (UniqueName: \"kubernetes.io/projected/4cda18ac-ae67-4bbd-be13-e386dbf09a34-kube-api-access-k8gr6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:03.271079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.271073    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cda18ac-ae67-4bbd-be13-e386dbf09a34-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:03.438724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.438679    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:03.952839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.952811    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"
Oct 30 12:22:03.953039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.952850    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87" event={"ID":"4cda18ac-ae67-4bbd-be13-e386dbf09a34","Type":"ContainerDied","Data":"1bb222518852f5c7658e01183354692d45c4f63dffd5ff31081adead8a84f135"}
Oct 30 12:22:03.953039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:03.952895    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb222518852f5c7658e01183354692d45c4f63dffd5ff31081adead8a84f135"
Oct 30 12:22:05.154065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:05.154027    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:06.961733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:06.961694    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:08.647669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:08.647622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:10.463399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:10.463370    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:11.010704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.010662    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"]
Oct 30 12:22:11.011916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.011890    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4cda18ac-ae67-4bbd-be13-e386dbf09a34" containerName="revoke-keys"
Oct 30 12:22:11.011916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.011919    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cda18ac-ae67-4bbd-be13-e386dbf09a34" containerName="revoke-keys"
Oct 30 12:22:11.012154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.012140    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cda18ac-ae67-4bbd-be13-e386dbf09a34" containerName="revoke-keys"
Oct 30 12:22:11.022571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.022541    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"]
Oct 30 12:22:11.022889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.022864    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.038471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.038426    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:22:11.050429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.050405    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:22:11.050589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.050427    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:22:11.050589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.050472    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:22:11.116615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.116573    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.116828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.116774    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.117065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.117034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs64l\" (UniqueName: \"kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.218643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.218603    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs64l\" (UniqueName: \"kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.218890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.218697    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.218890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.218844    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.219446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.219422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.221904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.221881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.242659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.242623    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs64l\" (UniqueName: \"kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l\") pod \"maas-api-revoke-sub-7ccc017f-vhhqf\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.349720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.349619    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:11.712568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:11.712106    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"]
Oct 30 12:22:11.717356 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:22:11.717330    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1e49c9_026c_459e_bd26_98cd5ec3a974.slice/crio-0b4f77f704c5bc21277fbeba84983ae6ec8e0887ce20c4e564c4112cadbdd88b WatchSource:0}: Error finding container 0b4f77f704c5bc21277fbeba84983ae6ec8e0887ce20c4e564c4112cadbdd88b: Status 404 returned error can't find the container with id 0b4f77f704c5bc21277fbeba84983ae6ec8e0887ce20c4e564c4112cadbdd88b
Oct 30 12:22:12.060042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:12.059952    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf" event={"ID":"bc1e49c9-026c-459e-bd26-98cd5ec3a974","Type":"ContainerStarted","Data":"0b4f77f704c5bc21277fbeba84983ae6ec8e0887ce20c4e564c4112cadbdd88b"}
Oct 30 12:22:12.356056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:12.350798    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:13.071410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:13.071375    5060 generic.go:356] "Generic (PLEG): container finished" podID="bc1e49c9-026c-459e-bd26-98cd5ec3a974" containerID="bacc8cc73f387416dacf3f9da26fb9280d97355ee10c4313456b7c7a805bbd48" exitCode=0
Oct 30 12:22:13.071832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:13.071487    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf" event={"ID":"bc1e49c9-026c-459e-bd26-98cd5ec3a974","Type":"ContainerDied","Data":"bacc8cc73f387416dacf3f9da26fb9280d97355ee10c4313456b7c7a805bbd48"}
Oct 30 12:22:14.125678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.125642    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:14.417979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.417950    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:14.473580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.473549    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l\" (UniqueName: \"kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l\") pod \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") "
Oct 30 12:22:14.473794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.473672    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca\") pod \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") "
Oct 30 12:22:14.473914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.473881    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca\") pod \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\" (UID: \"bc1e49c9-026c-459e-bd26-98cd5ec3a974\") "
Oct 30 12:22:14.474424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.474394    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca" pod "bc1e49c9-026c-459e-bd26-98cd5ec3a974" (UID: "bc1e49c9-026c-459e-bd26-98cd5ec3a974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:14.474755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.474731    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:14.476139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.476112    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca" pod "bc1e49c9-026c-459e-bd26-98cd5ec3a974" (UID: "bc1e49c9-026c-459e-bd26-98cd5ec3a974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:22:14.496477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.490742    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l" pod "bc1e49c9-026c-459e-bd26-98cd5ec3a974" (UID: "bc1e49c9-026c-459e-bd26-98cd5ec3a974"). InnerVolumeSpecName "kube-api-access-zs64l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:14.585479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.577014    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs64l\" (UniqueName: \"kubernetes.io/projected/bc1e49c9-026c-459e-bd26-98cd5ec3a974-kube-api-access-zs64l\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:14.585479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:14.577052    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1e49c9-026c-459e-bd26-98cd5ec3a974-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:15.166476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.163919    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf" event={"ID":"bc1e49c9-026c-459e-bd26-98cd5ec3a974","Type":"ContainerDied","Data":"0b4f77f704c5bc21277fbeba84983ae6ec8e0887ce20c4e564c4112cadbdd88b"}
Oct 30 12:22:15.166476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.163982    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b4f77f704c5bc21277fbeba84983ae6ec8e0887ce20c4e564c4112cadbdd88b"
Oct 30 12:22:15.166476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.164052    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"
Oct 30 12:22:15.252752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.252723    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:15.413975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.413942    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:15.811752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.811717    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:15.953018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:15.952980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:16.274136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.274097    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:16.769487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.768751    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"]
Oct 30 12:22:16.773477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.769917    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bc1e49c9-026c-459e-bd26-98cd5ec3a974" containerName="revoke-keys"
Oct 30 12:22:16.773477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.769945    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1e49c9-026c-459e-bd26-98cd5ec3a974" containerName="revoke-keys"
Oct 30 12:22:16.773477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.770191    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1e49c9-026c-459e-bd26-98cd5ec3a974" containerName="revoke-keys"
Oct 30 12:22:16.804869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.804830    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.805443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.805422    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"]
Oct 30 12:22:16.831778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.831746    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9b2407-openshift-default-dockercfg-96nbj\""
Oct 30 12:22:16.832731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.832583    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:16.960500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.890990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.984832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.984786    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.985556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.985522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.986133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.986111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.986354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.986319    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8zq\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.986580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.986563    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.986737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.986719    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.987004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.986972    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:16.987174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:16.987156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.083031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.081960    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:17.090690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090382    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.090690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090438    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.090690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090652    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.090690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8zq\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090740    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090766    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090854    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090886    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.090945    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.091383    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.091756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.091686    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.092338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.091930    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.095622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.095582    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.096576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.096538    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.113623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.109173    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.113623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.111826    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.135737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.135708    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8zq\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.141298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.141273    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token\") pod \"e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") " pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.260199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.260161    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:17.616300 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:22:17.616265    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693bff5e_b1b9_4022_9dff_706c5a07bc07.slice/crio-23dc59a80e1d670fc00bade24998b26961888d38d8dd30acab36b6e1be63d7b8 WatchSource:0}: Error finding container 23dc59a80e1d670fc00bade24998b26961888d38d8dd30acab36b6e1be63d7b8: Status 404 returned error can't find the container with id 23dc59a80e1d670fc00bade24998b26961888d38d8dd30acab36b6e1be63d7b8
Oct 30 12:22:17.620209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.620166    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:17.620335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.620256    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:17.620335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.620299    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:17.620477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.620375    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"]
Oct 30 12:22:17.971275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:17.971231    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:18.197235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:18.197085    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" event={"ID":"693bff5e-b1b9-4022-9dff-706c5a07bc07","Type":"ContainerStarted","Data":"c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"}
Oct 30 12:22:18.197235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:18.197174    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" event={"ID":"693bff5e-b1b9-4022-9dff-706c5a07bc07","Type":"ContainerStarted","Data":"23dc59a80e1d670fc00bade24998b26961888d38d8dd30acab36b6e1be63d7b8"}
Oct 30 12:22:18.248402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:18.247415    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podStartSLOduration=2.247399518 podStartE2EDuration="2.247399518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:18.246115048 +0000 UTC m=+3142.040999552" watchObservedRunningTime="2026-10-30 12:22:18.247399518 +0000 UTC m=+3142.042284012"
Oct 30 12:22:18.260708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:18.260664    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:18.262643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:18.262609    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:22:18.262808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:18.262670    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:22:19.268092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:19.268064    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:19.857863 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:22:19.857825    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:22:20.037087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:20.037038    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:20.220762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:20.220726    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:20.221919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:20.221895    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:22:20.682722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:20.682632    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:20.851641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:20.851591    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:21.232114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.232080    5060 generic.go:356] "Generic (PLEG): container finished" podID="98605626-4680-4b6c-a7fc-59b59c37d854" containerID="83c3140f31d0abdce33683c9ba67b12b2becc7336770d3129f7beb8340cce8a3" exitCode=0
Oct 30 12:22:21.232313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.232152    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" event={"ID":"98605626-4680-4b6c-a7fc-59b59c37d854","Type":"ContainerDied","Data":"83c3140f31d0abdce33683c9ba67b12b2becc7336770d3129f7beb8340cce8a3"}
Oct 30 12:22:21.232623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.232607    5060 scope.go:122] "RemoveContainer" containerID="83c3140f31d0abdce33683c9ba67b12b2becc7336770d3129f7beb8340cce8a3"
Oct 30 12:22:21.234445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.234418    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:21.234590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.234530    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:21.234590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.234566    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:21.364623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.364549    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:21.501326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:21.501225    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:22.017385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.017344    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:22:22.092474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.084307    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:22.273664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.273572    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" event={"ID":"98605626-4680-4b6c-a7fc-59b59c37d854","Type":"ContainerStarted","Data":"688f88e5a5785831a07073230b8c992dd928e6efde749d55defaac7b81943642"}
Oct 30 12:22:22.313480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.297413    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:22.442064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.442022    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:22.604531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.604008    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:22.654478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.646540    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:22:22.688483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.675575    5060 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 30 12:22:22.688483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:22.675658    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 30 12:22:23.428479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:23.427539    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:23.599217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:23.599178    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:23.650859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:23.650824    5060 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 30 12:22:23.651143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:23.651116    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 30 12:22:24.022139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:24.022104    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:24.214596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:24.214561    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:24.645816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:24.645777    5060 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 30 12:22:24.646281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:24.645842    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 30 12:22:24.667942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:24.667904    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"]
Oct 30 12:22:24.789495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:24.784190    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:25.645400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:25.645363    5060 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 30 12:22:25.645630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:25.645415    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 30 12:22:26.017143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:26.017105    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:26.645218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:26.645179    5060 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 30 12:22:26.645468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:26.645242    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 30 12:22:27.644811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:27.644767    5060 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 30 12:22:27.645218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:27.644827    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 30 12:22:27.921307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:27.921195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:28.650904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:28.650868    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:22:28.651411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:28.651394    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:22:28.652343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:28.652322    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:22:29.668485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.668409    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm" podUID="08cf6517-b8f4-4324-b9ca-07308b9e1297" containerName="payload-processing" containerID="cri-o://37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b" gracePeriod=25
Oct 30 12:22:29.742583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.742548    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:29.968793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.968766    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:22:29.998373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.998345    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume\") pod \"08cf6517-b8f4-4324-b9ca-07308b9e1297\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") "
Oct 30 12:22:29.998551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.998520    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials\") pod \"08cf6517-b8f4-4324-b9ca-07308b9e1297\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") "
Oct 30 12:22:29.998604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.998585    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume\") pod \"08cf6517-b8f4-4324-b9ca-07308b9e1297\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") "
Oct 30 12:22:29.998757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.998726    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c\") pod \"08cf6517-b8f4-4324-b9ca-07308b9e1297\" (UID: \"08cf6517-b8f4-4324-b9ca-07308b9e1297\") "
Oct 30 12:22:29.998757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.998733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume" pod "08cf6517-b8f4-4324-b9ca-07308b9e1297" (UID: "08cf6517-b8f4-4324-b9ca-07308b9e1297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:29.998979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.998956    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume" pod "08cf6517-b8f4-4324-b9ca-07308b9e1297" (UID: "08cf6517-b8f4-4324-b9ca-07308b9e1297"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:29.999399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.999377    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:29.999504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:29.999404    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08cf6517-b8f4-4324-b9ca-07308b9e1297-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:30.001006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.000969    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials" pod "08cf6517-b8f4-4324-b9ca-07308b9e1297" (UID: "08cf6517-b8f4-4324-b9ca-07308b9e1297"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:30.001115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.001013    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c" pod "08cf6517-b8f4-4324-b9ca-07308b9e1297" (UID: "08cf6517-b8f4-4324-b9ca-07308b9e1297"). InnerVolumeSpecName "kube-api-access-d9t4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:30.092628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.092584    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:30.100601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.100575    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:30.100601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.100599    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9t4c\" (UniqueName: \"kubernetes.io/projected/08cf6517-b8f4-4324-b9ca-07308b9e1297-kube-api-access-d9t4c\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:22:30.238505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.238469    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:30.390558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.390524    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:30.400170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.400142    5060 generic.go:356] "Generic (PLEG): container finished" podID="08cf6517-b8f4-4324-b9ca-07308b9e1297" containerID="37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b" exitCode=0
Oct 30 12:22:30.400343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.400188    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"
Oct 30 12:22:30.400343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.400208    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm" event={"ID":"08cf6517-b8f4-4324-b9ca-07308b9e1297","Type":"ContainerDied","Data":"37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b"}
Oct 30 12:22:30.400343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.400275    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm" event={"ID":"08cf6517-b8f4-4324-b9ca-07308b9e1297","Type":"ContainerDied","Data":"88993f2413ccfe637b97fa0afcf5ace3d92374203aa43a4f2a88908240abca69"}
Oct 30 12:22:30.400343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.400305    5060 scope.go:122] "RemoveContainer" containerID="37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b"
Oct 30 12:22:30.416880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.416856    5060 scope.go:122] "RemoveContainer" containerID="37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b"
Oct 30 12:22:30.417220 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:22:30.417196    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b\": container with ID starting with 37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b not found: ID does not exist" containerID="37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b"
Oct 30 12:22:30.417314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.417230    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b"} err="failed to get container status \"37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b\": rpc error: code = NotFound desc = could not find container \"37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b\": container with ID starting with 37387d90a70f72dfe23f248248aff256e28d96087b1679fbbdb17b473829185b not found: ID does not exist"
Oct 30 12:22:30.456707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.456668    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"]
Oct 30 12:22:30.462804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:30.462772    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-6784bsrrm"]
Oct 30 12:22:31.786344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:31.786306    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:32.336331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:32.336301    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cf6517-b8f4-4324-b9ca-07308b9e1297" path="/var/lib/kubelet/pods/08cf6517-b8f4-4324-b9ca-07308b9e1297/volumes"
Oct 30 12:22:33.656139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:33.645919    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:35.094752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.094710    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:35.461067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.461030    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:35.601577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.601538    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:35.997714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.997673    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"]
Oct 30 12:22:35.998865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.998844    5060 cpu_manager.go:414] "containerMap: removing container" podUID="08cf6517-b8f4-4324-b9ca-07308b9e1297" containerName="payload-processing"
Oct 30 12:22:35.998944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.998869    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cf6517-b8f4-4324-b9ca-07308b9e1297" containerName="payload-processing"
Oct 30 12:22:35.999094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:35.999081    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cf6517-b8f4-4324-b9ca-07308b9e1297" containerName="payload-processing"
Oct 30 12:22:36.005074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.005050    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.032400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.032275    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9d71fa-openshift-default-dockercfg-d62s2\""
Oct 30 12:22:36.038286 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.038260    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"]
Oct 30 12:22:36.090052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.090019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:36.095333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095306    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.095795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095344    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcv9\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.095795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095437    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.095795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095481    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.095795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095506    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.095795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095719    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.096062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095802    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.096062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095838    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.096062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.095935    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199494    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199552    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199720    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199776    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199830    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.199911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.200067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.200480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.200101    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcv9\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.219608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.215907    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.219608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.216181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.219608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.216431    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.219608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.216997    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.232481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.229581    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.263666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.248149    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcv9\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.263666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.258514    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.269354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.269281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.272156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.272128    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:36.300476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.296770    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.351607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.349324    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:36.783180 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:22:36.783140    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3403dde0_012d_447a_a9a2_0516569d1770.slice/crio-19872d3c213f46f42c865390f7f6f2e5e97cbe443f544a7c93f86b8561a81a5a WatchSource:0}: Error finding container 19872d3c213f46f42c865390f7f6f2e5e97cbe443f544a7c93f86b8561a81a5a: Status 404 returned error can't find the container with id 19872d3c213f46f42c865390f7f6f2e5e97cbe443f544a7c93f86b8561a81a5a
Oct 30 12:22:36.787045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.786526    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:36.787045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.786614    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:36.787045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.786654    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:36.787045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.786867    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"]
Oct 30 12:22:36.995761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:36.995709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:37.150207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:37.150105    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:37.361428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:37.361385    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:37.521185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:37.521151    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" event={"ID":"3403dde0-012d-447a-a9a2-0516569d1770","Type":"ContainerStarted","Data":"2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"}
Oct 30 12:22:37.521384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:37.521209    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" event={"ID":"3403dde0-012d-447a-a9a2-0516569d1770","Type":"ContainerStarted","Data":"19872d3c213f46f42c865390f7f6f2e5e97cbe443f544a7c93f86b8561a81a5a"}
Oct 30 12:22:37.592436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:37.592372    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podStartSLOduration=2.592357681 podStartE2EDuration="2.592357681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:37.579476881 +0000 UTC m=+3161.374361375" watchObservedRunningTime="2026-10-30 12:22:37.592357681 +0000 UTC m=+3161.387242174"
Oct 30 12:22:37.962371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:37.962332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:38.108850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:38.108818    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:38.380532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:38.379770    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:38.545384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:38.544576    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:39.384630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:39.384586    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:39.545483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:39.541900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:39.622515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:39.608277    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:39.622515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:39.609690    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:22:40.111612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:40.111575    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:40.270421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:40.270375    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:41.366926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:41.366890    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:41.874180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:41.874132    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:42.018662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:42.018622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:42.056625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:42.056575    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:22:42.056848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:42.056683    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mnfsj" podUID="f26c00f4-7dcf-42fc-bf26-bbaddc9ea873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:22:43.240284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:43.240242    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:44.237475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:44.236535    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:44.381338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:44.381297    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:45.244634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:45.244596    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:45.380919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:45.380864    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:46.367605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:46.367563    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:47.193907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:47.193866    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:48.966012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:48.965974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:49.858903 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:22:49.858866    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:22:50.752921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:50.752882    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:51.967416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:51.967048    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:52.105906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:52.105856    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:52.859864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:52.859827    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:53.544971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:53.544931    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:53.704477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:53.702107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:54.049904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.049863    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"]
Oct 30 12:22:54.073831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.073797    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.088391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.088357    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"]
Oct 30 12:22:54.098248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.097288    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ae3538-openshift-default-dockercfg-kzncb\""
Oct 30 12:22:54.135764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.135720    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.135764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.135775    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.135916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr52s\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.135954    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.136055    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.136108    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.136201    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.136240    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.136275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.136266    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237487    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237553    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237657    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237740    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237812    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237847    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.237979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.237957    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr52s\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.238244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.238212    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.238368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.238240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.238486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.238446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.238684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.238658    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.238771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.238727    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.240646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.240624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.240881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.240860    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.296476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.275511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:54.296476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.294293    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr52s\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.319101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.319024    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.414473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.414419    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:54.473141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:54.473104    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:55.049038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:55.048964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:55.226281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:55.226162    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"]
Oct 30 12:22:55.233858 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:22:55.233817    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e7b99b_42cc_48c7_806b_ed940feee094.slice/crio-421d47d3c30f8af86bdc7e5424cb829d6c97950cc14a1a82034161de90cb2f52 WatchSource:0}: Error finding container 421d47d3c30f8af86bdc7e5424cb829d6c97950cc14a1a82034161de90cb2f52: Status 404 returned error can't find the container with id 421d47d3c30f8af86bdc7e5424cb829d6c97950cc14a1a82034161de90cb2f52
Oct 30 12:22:55.239015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:55.238969    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:55.239136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:55.239091    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:55.239217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:55.239138    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:22:56.135479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:56.134569    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" event={"ID":"33e7b99b-42cc-48c7-806b-ed940feee094","Type":"ContainerStarted","Data":"d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"}
Oct 30 12:22:56.135479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:56.134640    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" event={"ID":"33e7b99b-42cc-48c7-806b-ed940feee094","Type":"ContainerStarted","Data":"421d47d3c30f8af86bdc7e5424cb829d6c97950cc14a1a82034161de90cb2f52"}
Oct 30 12:22:56.209520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:56.202529    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podStartSLOduration=2.202492936 podStartE2EDuration="2.202492936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:56.199757461 +0000 UTC m=+3179.994641955" watchObservedRunningTime="2026-10-30 12:22:56.202492936 +0000 UTC m=+3179.997377431"
Oct 30 12:22:56.420760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:56.420660    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:56.437215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:56.436814    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:57.092172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:57.092136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:57.193059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:57.193028    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:57.212251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:57.211911    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:22:58.630646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:58.630610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:58.783806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:58.783772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:59.697124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:59.697086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:22:59.832736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:59.832695    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 12:22:59.895709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:22:59.895667    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:23:00.622483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:00.622398    5060 scope.go:122] "RemoveContainer" containerID="3af03781c107ee84b6231b6375ab7f504e760a0174e4a0291175310320e9c6ea"
Oct 30 12:23:01.308183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:01.308149    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:01.462696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:01.462663    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:01.704474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:01.702606    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:03.581189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:03.581150    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:05.398715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:05.398672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:07.108350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:07.108254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:08.948267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:08.948215    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:10.792845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:10.792797    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:10.946310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:10.946269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:12.789318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:12.789274    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:14.651970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:14.651926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:16.342741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:16.342705    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:18.197502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:18.195080    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:19.858435 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:23:19.858388    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:23:19.948997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:19.948957    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:21.704881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:21.704845    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:23.457717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:23.457673    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:25.144955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:25.144914    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:26.839102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:26.839072    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:28.702066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:28.702021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:30.485705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:30.485661    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:30.641904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:30.641863    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:30.849936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:30.849843    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:32.644346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:32.644317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:34.447941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:34.447905    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:35.032248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.032217    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"]
Oct 30 12:23:35.044714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.044684    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc8e25f-hrj7s"]
Oct 30 12:23:35.708907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.708880    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:35.860490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.860053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:35.968069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.967985    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"]
Oct 30 12:23:35.972984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.972959    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:35.993790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:35.993751    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-5cdfef\"/\"openshift-service-ca.crt\""
Oct 30 12:23:36.006868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.006832    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-5cdfef\"/\"e2e-usage-sim-dc1a34-cert\""
Oct 30 12:23:36.007476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.007434    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-5cdfef\"/\"default-dockercfg-kv2cw\""
Oct 30 12:23:36.007698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.007676    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-5cdfef\"/\"kube-root-ca.crt\""
Oct 30 12:23:36.018404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.018375    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"]
Oct 30 12:23:36.057154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.057121    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls\") pod \"e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") " pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.057348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.057315    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rxw\" (UniqueName: \"kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw\") pod \"e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") " pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.159295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.159258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls\") pod \"e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") " pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.159550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.159446    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rxw\" (UniqueName: \"kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw\") pod \"e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") " pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.162170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.162144    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls\") pod \"e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") " pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.201358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.201311    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rxw\" (UniqueName: \"kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw\") pod \"e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") " pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.232821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.232748    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:36.318639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.317187    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:36.338515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.338486    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e15960-72b5-4465-aa2a-04deb83f7873" path="/var/lib/kubelet/pods/f6e15960-72b5-4465-aa2a-04deb83f7873/volumes"
Oct 30 12:23:36.670366 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:23:36.670332    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aef7f68_0ea1_4401_bdf3_4206d5244009.slice/crio-b77b790665f667c679b4f843a14eb3efbf0652b97a40ff883c6bf63d8569b32b WatchSource:0}: Error finding container b77b790665f667c679b4f843a14eb3efbf0652b97a40ff883c6bf63d8569b32b: Status 404 returned error can't find the container with id b77b790665f667c679b4f843a14eb3efbf0652b97a40ff883c6bf63d8569b32b
Oct 30 12:23:36.673062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:36.673036    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"]
Oct 30 12:23:37.676205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:37.676164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj" event={"ID":"9aef7f68-0ea1-4401-bdf3-4206d5244009","Type":"ContainerStarted","Data":"9fe13b942c6d5c36c09771e24202c3ee7d67b97ba8a76793db685fedef93e062"}
Oct 30 12:23:37.676788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:37.676213    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj" event={"ID":"9aef7f68-0ea1-4401-bdf3-4206d5244009","Type":"ContainerStarted","Data":"b77b790665f667c679b4f843a14eb3efbf0652b97a40ff883c6bf63d8569b32b"}
Oct 30 12:23:37.676788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:37.676367    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:37.713200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:37.713150    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:23:37.725067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:37.724992    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj" podStartSLOduration=2.724973898 podStartE2EDuration="2.724973898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:37.71785273 +0000 UTC m=+3221.512737224" watchObservedRunningTime="2026-10-30 12:23:37.724973898 +0000 UTC m=+3221.519858393"
Oct 30 12:23:38.094074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:38.094044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:39.958911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:39.948321    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:41.623160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:41.623123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:42.001645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.001527    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:42.149897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.149793    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:42.778502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.775561    5060 generic.go:356] "Generic (PLEG): container finished" podID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerID="c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67" exitCode=0
Oct 30 12:23:42.778502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.775734    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" event={"ID":"693bff5e-b1b9-4022-9dff-706c5a07bc07","Type":"ContainerDied","Data":"c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"}
Oct 30 12:23:42.778502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.776365    5060 scope.go:122] "RemoveContainer" containerID="c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"
Oct 30 12:23:42.791480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.787683    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:23:42.791480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.787776    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:23:42.791480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:42.787820    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:23:43.230283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:43.214734    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded"
Oct 30 12:23:43.242513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:43.241957    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:23:43.791211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:43.791170    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" event={"ID":"693bff5e-b1b9-4022-9dff-706c5a07bc07","Type":"ContainerStarted","Data":"c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695"}
Oct 30 12:23:43.934322 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:43.933758    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:44.034722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:44.034681    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"]
Oct 30 12:23:44.039360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:44.039330    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9919bd-mdx6c"]
Oct 30 12:23:44.260494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:44.260447    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:23:44.260698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:44.260673    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:23:44.260751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:44.260722    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:23:44.338102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:44.338068    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee09daa-64d0-402e-aafc-a3986d1f8e82" path="/var/lib/kubelet/pods/7ee09daa-64d0-402e-aafc-a3986d1f8e82/volumes"
Oct 30 12:23:45.260904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:45.260850    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:23:45.261344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:45.260921    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:23:45.741071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:45.741032    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:46.260876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:46.260836    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:23:46.261089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:46.260889    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:23:47.034373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:47.034335    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"]
Oct 30 12:23:47.046512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:47.046478    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f95da1-vhtsl"]
Oct 30 12:23:47.261100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:47.261066    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:23:47.261525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:47.261122    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:23:47.558574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:47.558522    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:48.260935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:48.260890    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:23:48.261132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:48.260962    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:23:48.337799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:48.337763    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7f05bb-b672-486d-8870-1eeeb1dbdc71" path="/var/lib/kubelet/pods/ea7f05bb-b672-486d-8870-1eeeb1dbdc71/volumes"
Oct 30 12:23:49.261432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:49.261395    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 12:23:49.262053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:49.261581    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 12:23:49.262425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:49.262400    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:49.859939 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:23:49.859896    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:23:50.220647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:50.220603    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:23:50.266338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:50.266304    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:23:50.893315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:50.893284    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:23:50.943120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:50.943086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:51.031758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:51.031723    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"]
Oct 30 12:23:51.036768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:51.036741    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedc5ba7-dlwql"]
Oct 30 12:23:51.115084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:51.115032    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:52.337491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:52.337439    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13430e6c-b863-43ce-9459-fb4042803794" path="/var/lib/kubelet/pods/13430e6c-b863-43ce-9459-fb4042803794/volumes"
Oct 30 12:23:52.955001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:52.954963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:53.685751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:53.685709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:54.225938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:54.225900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:54.756350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:54.756311    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:55.427550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:55.427518    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"]
Oct 30 12:23:56.480068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:56.480033    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:58.235500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:58.235429    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:23:59.033008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:59.032954    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"]
Oct 30 12:23:59.044892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:23:59.044863    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9184d92-4k6d2"]
Oct 30 12:24:00.083795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.083750    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:00.336942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.336858    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eab30c0-09cd-4b8b-ab48-36427b5fbf7b" path="/var/lib/kubelet/pods/5eab30c0-09cd-4b8b-ab48-36427b5fbf7b/volumes"
Oct 30 12:24:00.428053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.427998    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw" podUID="39b9c0d2-88eb-4b96-8dcd-249d153c11c8" containerName="payload-processing" containerID="cri-o://1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e" gracePeriod=25
Oct 30 12:24:00.519864 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:24:00.519821    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b9c0d2_88eb_4b96_8dcd_249d153c11c8.slice/crio-conmon-1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b9c0d2_88eb_4b96_8dcd_249d153c11c8.slice/crio-1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 12:24:00.749128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.749101    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:24:00.853868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.853835    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4\") pod \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") "
Oct 30 12:24:00.854168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.853903    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume\") pod \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") "
Oct 30 12:24:00.854168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.853942    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume\") pod \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") "
Oct 30 12:24:00.854168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.854054    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials\") pod \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\" (UID: \"39b9c0d2-88eb-4b96-8dcd-249d153c11c8\") "
Oct 30 12:24:00.854671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.854347    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume" pod "39b9c0d2-88eb-4b96-8dcd-249d153c11c8" (UID: "39b9c0d2-88eb-4b96-8dcd-249d153c11c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:24:00.854671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.854411    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume" pod "39b9c0d2-88eb-4b96-8dcd-249d153c11c8" (UID: "39b9c0d2-88eb-4b96-8dcd-249d153c11c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:24:00.857246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.856354    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:24:00.857246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.856386    5060 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-routing-overlay-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:24:00.857246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.856639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4" pod "39b9c0d2-88eb-4b96-8dcd-249d153c11c8" (UID: "39b9c0d2-88eb-4b96-8dcd-249d153c11c8"). InnerVolumeSpecName "kube-api-access-q48r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:24:00.869096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.868979    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials" pod "39b9c0d2-88eb-4b96-8dcd-249d153c11c8" (UID: "39b9c0d2-88eb-4b96-8dcd-249d153c11c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:24:00.957911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.957874    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q48r4\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-kube-api-access-q48r4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:24:00.957911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.957908    5060 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39b9c0d2-88eb-4b96-8dcd-249d153c11c8-provider-credentials\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:24:00.993699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.993662    5060 generic.go:356] "Generic (PLEG): container finished" podID="39b9c0d2-88eb-4b96-8dcd-249d153c11c8" containerID="1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e" exitCode=0
Oct 30 12:24:00.993878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.993756    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"
Oct 30 12:24:00.993878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.993756    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw" event={"ID":"39b9c0d2-88eb-4b96-8dcd-249d153c11c8","Type":"ContainerDied","Data":"1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e"}
Oct 30 12:24:00.993971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.993932    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw" event={"ID":"39b9c0d2-88eb-4b96-8dcd-249d153c11c8","Type":"ContainerDied","Data":"29b0f48efec0ab900cce40da9da644d445cd38f37b67da85c700f2f0838a45e0"}
Oct 30 12:24:00.993971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:00.993964    5060 scope.go:122] "RemoveContainer" containerID="1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e"
Oct 30 12:24:01.007148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:01.007127    5060 scope.go:122] "RemoveContainer" containerID="1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e"
Oct 30 12:24:01.007500 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:24:01.007472    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e\": container with ID starting with 1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e not found: ID does not exist" containerID="1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e"
Oct 30 12:24:01.007592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:01.007511    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e"} err="failed to get container status \"1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e\": rpc error: code = NotFound desc = could not find container \"1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e\": container with ID starting with 1dbf4d65bff98b1ff724c0cd4366bc63d7d58719a34e3ad08528940d78801a3e not found: ID does not exist"
Oct 30 12:24:01.048034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:01.047999    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"]
Oct 30 12:24:01.053804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:01.053774    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cxp5qw"]
Oct 30 12:24:01.856812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:01.856771    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:02.337125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:02.337092    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b9c0d2-88eb-4b96-8dcd-249d153c11c8" path="/var/lib/kubelet/pods/39b9c0d2-88eb-4b96-8dcd-249d153c11c8/volumes"
Oct 30 12:24:03.626417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:03.626381    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:05.375326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:05.375287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:07.334758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:07.334720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:09.107385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:09.107336    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:10.882943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:10.882880    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:12.757303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:12.757248    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:14.646111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:14.646068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:16.333329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:16.333285    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:18.018946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:18.018817    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:18.247968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:18.247931    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:18.256650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:18.256616    5060 generic.go:356] "Generic (PLEG): container finished" podID="c85b9afe-dc24-4cf9-8639-a59f727c4bb2" containerID="9c5555ec6b0a6a863563a651566c4578a9baa32a8ebc78f8501cbd1b6bead830" exitCode=1
Oct 30 12:24:18.256838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:18.256697    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph" event={"ID":"c85b9afe-dc24-4cf9-8639-a59f727c4bb2","Type":"ContainerDied","Data":"9c5555ec6b0a6a863563a651566c4578a9baa32a8ebc78f8501cbd1b6bead830"}
Oct 30 12:24:18.257315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:18.257293    5060 scope.go:122] "RemoveContainer" containerID="9c5555ec6b0a6a863563a651566c4578a9baa32a8ebc78f8501cbd1b6bead830"
Oct 30 12:24:19.269774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:19.269737    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph" event={"ID":"c85b9afe-dc24-4cf9-8639-a59f727c4bb2","Type":"ContainerStarted","Data":"0c5cd1d77f91b15cdefb3dddd101a0a711cff4d0261c7a1fa636d4d61d15170d"}
Oct 30 12:24:19.270226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:19.269973    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 12:24:19.867036 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:24:19.866992    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:24:19.890482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:19.879329    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:21.365420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.365375    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"]
Oct 30 12:24:21.366742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.366713    5060 cpu_manager.go:414] "containerMap: removing container" podUID="39b9c0d2-88eb-4b96-8dcd-249d153c11c8" containerName="payload-processing"
Oct 30 12:24:21.366742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.366742    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b9c0d2-88eb-4b96-8dcd-249d153c11c8" containerName="payload-processing"
Oct 30 12:24:21.367004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.366988    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b9c0d2-88eb-4b96-8dcd-249d153c11c8" containerName="payload-processing"
Oct 30 12:24:21.374634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.371849    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.406086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.406042    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"]
Oct 30 12:24:21.547463 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.547400    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.547685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.547603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chx4g\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.547779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.547719    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.547915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.547885    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.548064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.547957    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.548064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.548034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.548064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.548062    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.548263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.548240    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.548323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.548284    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650117    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650168    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chx4g\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650214    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650343    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650370    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650425    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650474    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650572    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650600    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.650882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650794    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.651011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.650973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.651076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.651009    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.651262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.651240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.651918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.651890    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.653392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.653364    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.654018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.653990    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.681924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.681856    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.682105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.682034    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chx4g\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.702403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.702347    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:21.793048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:21.792937    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:22.082563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.082519    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"]
Oct 30 12:24:22.095356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.095312    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:22.095588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.095414    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:22.095588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.095505    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:22.307478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.306528    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" event={"ID":"5f872375-b24f-47cf-984e-06b1e94b9b2c","Type":"ContainerStarted","Data":"e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d"}
Oct 30 12:24:22.307478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.306590    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" event={"ID":"5f872375-b24f-47cf-984e-06b1e94b9b2c","Type":"ContainerStarted","Data":"8d3796516c24d93c2d4f6e92c53cb5a0fe4657f4450d261d5efcd94cbb09632b"}
Oct 30 12:24:22.356830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.356718    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podStartSLOduration=1.356696433 podStartE2EDuration="1.356696433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:22.355771946 +0000 UTC m=+3266.150656441" watchObservedRunningTime="2026-10-30 12:24:22.356696433 +0000 UTC m=+3266.151580928"
Oct 30 12:24:22.494088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.490199    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"]
Oct 30 12:24:22.506649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.506609    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.525950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.524858    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a1f322\"/\"openshift-service-ca.crt\""
Oct 30 12:24:22.525950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.525715    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a1f322\"/\"test-model-a1f322-kserve-self-signed-certs\""
Oct 30 12:24:22.525950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.525816    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a1f322\"/\"default-dockercfg-twkvg\""
Oct 30 12:24:22.525950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.525899    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a1f322\"/\"kube-root-ca.crt\""
Oct 30 12:24:22.534301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.534191    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"]
Oct 30 12:24:22.594480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.577539    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:22.670261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.670125    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.670261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.670185    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmpd\" (UniqueName: \"kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.670565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.670348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.670565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.670411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.670565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.670468    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.671080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.670992    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.705670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.705569    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:22.776149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.776068    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.776619    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.776714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.776745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmpd\" (UniqueName: \"kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.776921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.776985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.777029    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.779486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.777559    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.787495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.784157    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.845494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.837543    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.845494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.837884    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:22.845494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:22.844323    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmpd\" (UniqueName: \"kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd\") pod \"test-model-a1f322-kserve-65558fb68f-wzts9\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") " pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:23.142263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.142229    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:23.324044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.323953    5060 generic.go:356] "Generic (PLEG): container finished" podID="33e7b99b-42cc-48c7-806b-ed940feee094" containerID="d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f" exitCode=0
Oct 30 12:24:23.324044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.324031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" event={"ID":"33e7b99b-42cc-48c7-806b-ed940feee094","Type":"ContainerDied","Data":"d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"}
Oct 30 12:24:23.324662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.324634    5060 scope.go:122] "RemoveContainer" containerID="d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"
Oct 30 12:24:23.326732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.326703    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:23.326857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.326770    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:23.326857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.326801    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:23.504852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.503601    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"]
Oct 30 12:24:23.675282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.675242    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:23.715474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.715204    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:23.715474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:23.715312    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:24.337617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.337577    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" event={"ID":"1f86dd20-d255-4a9e-baa2-5dd9521bb79c","Type":"ContainerStarted","Data":"4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab"}
Oct 30 12:24:24.337617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.337620    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" event={"ID":"1f86dd20-d255-4a9e-baa2-5dd9521bb79c","Type":"ContainerStarted","Data":"67a9abe450075e519dea53dc2e0817831ede797229e53349c7d808ccba464c3a"}
Oct 30 12:24:24.343111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.343068    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" event={"ID":"33e7b99b-42cc-48c7-806b-ed940feee094","Type":"ContainerStarted","Data":"ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56"}
Oct 30 12:24:24.397522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.397433    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" podStartSLOduration=2.39741087 podStartE2EDuration="2.39741087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:24.37995249 +0000 UTC m=+3268.174836983" watchObservedRunningTime="2026-10-30 12:24:24.39741087 +0000 UTC m=+3268.192295364"
Oct 30 12:24:24.420489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.418885    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:24:24.420489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.419025    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:24.420489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.419084    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:24.613504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.612880    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:24.613504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.612934    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:24.706491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.702909    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:24.706491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:24.702990    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:25.146604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.146540    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:25.404539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.401253    5060 generic.go:356] "Generic (PLEG): container finished" podID="3403dde0-012d-447a-a9a2-0516569d1770" containerID="2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80" exitCode=0
Oct 30 12:24:25.404539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.402558    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" event={"ID":"3403dde0-012d-447a-a9a2-0516569d1770","Type":"ContainerDied","Data":"2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"}
Oct 30 12:24:25.404539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.403015    5060 scope.go:122] "RemoveContainer" containerID="2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"
Oct 30 12:24:25.417482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.416435    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:25.417482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.416571    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:25.417482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.416626    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:25.418386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.417782    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:25.418386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.417828    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:25.645095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.644889    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:25.771957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.755657    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:25.772309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:25.772266    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:26.434480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:26.429269    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:26.434480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:26.429341    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:26.463094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:26.463063    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" event={"ID":"3403dde0-012d-447a-a9a2-0516569d1770","Type":"ContainerStarted","Data":"71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e"}
Oct 30 12:24:26.711412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:26.711373    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:26.712084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:26.712053    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:27.197605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.197567    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:24:27.355852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.355820    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:24:27.356223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.356193    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:27.356423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.356399    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:27.423907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.423867    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:27.424208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.424178    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:27.467394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.467303    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:27.633553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.626484    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:27.710482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.710109    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:27.710482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.710181    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:27.774532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:27.774098    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:28.435479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:28.434559    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:28.435479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:28.434621    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:28.435479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:28.435377    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:28.435479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:28.435418    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:28.750492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:28.745656    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:28.750492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:28.745718    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:29.362560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.362518    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:29.363173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.363138    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:29.417075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.417036    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:29.440665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.440627    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:29.444688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.444641    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:29.503609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.503559    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:24:29.529602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.529551    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:24:29.703915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.703876    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 12:24:29.704217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.704190    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 12:24:29.830480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.821043    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 12:24:29.830480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:29.823539    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:24:30.296477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.296042    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-h4nph"
Oct 30 12:24:30.355871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.355835    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:30.356143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.356115    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:30.417026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.416987    5060 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 30 12:24:30.417751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.417717    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 30 12:24:30.849198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.836827    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:30.849198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.838656    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:30.856791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:30.856757    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:24:31.129166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:31.129077    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:31.356207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:31.356175    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:31.356395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:31.356240    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:31.430126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:31.430040    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:24:31.432195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:31.432176    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:24:32.355703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:32.355671    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 12:24:32.355879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:32.355721    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 12:24:32.971859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:32.971817    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:33.143495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:33.143436    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:33.156858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:33.156830    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:33.360676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:33.360586    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:24:33.361028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:33.360994    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:24:33.361582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:33.361564    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:24:33.861646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:33.861609    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:24:34.747035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:34.746993    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:36.568396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:36.568283    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:38.271749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:38.271693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:40.061771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:40.059696    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:41.827495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:41.826668    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:43.638273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:43.638233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:45.561076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:45.561033    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:46.033624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.033588    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"]
Oct 30 12:24:46.045712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.045676    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e97f78-hsnmc"]
Oct 30 12:24:46.225472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.221704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:46.229878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.229846    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"]
Oct 30 12:24:46.237944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.235804    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.256332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.256294    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"]
Oct 30 12:24:46.264424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.264391    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9b2407\"/\"default-dockercfg-fcsc2\""
Oct 30 12:24:46.264652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.264566    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9b2407\"/\"openshift-service-ca.crt\""
Oct 30 12:24:46.264742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.264681    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9b2407\"/\"kube-root-ca.crt\""
Oct 30 12:24:46.264810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.264801    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9b2407\"/\"test-model-9b2407-kserve-self-signed-certs\""
Oct 30 12:24:46.277312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.277275    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.277655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.277631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.278009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.277985    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndl7\" (UniqueName: \"kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.278214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.278195    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.278413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.278395    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.278604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.278584    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.342206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.342110    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9781c203-4f34-4803-98a4-ea446780d716" path="/var/lib/kubelet/pods/9781c203-4f34-4803-98a4-ea446780d716/volumes"
Oct 30 12:24:46.380374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.380338    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndl7\" (UniqueName: \"kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.380624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.380483    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.380624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.380570    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.380796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.380692    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.380869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.380802    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.380934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.380872    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.381367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.381324    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.381367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.381347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.381806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.381784    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.384742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.384719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.385074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.385052    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:46.387045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.387021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.405719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.405687    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndl7\" (UniqueName: \"kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7\") pod \"test-model-9b2407-kserve-6846698459-4cwpv\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") " pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.571047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.570751    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:46.724129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:46.724090    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:47.010830 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:24:47.010799    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9352f0ab_4592_4083_a373_eb5780165613.slice/crio-dcb4bb72002fc9c94d7b15cc3cd1af6fd48299ba49a6c585ea42407b064b26f0 WatchSource:0}: Error finding container dcb4bb72002fc9c94d7b15cc3cd1af6fd48299ba49a6c585ea42407b064b26f0: Status 404 returned error can't find the container with id dcb4bb72002fc9c94d7b15cc3cd1af6fd48299ba49a6c585ea42407b064b26f0
Oct 30 12:24:47.013499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:47.013474    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"]
Oct 30 12:24:47.067988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:47.067957    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"]
Oct 30 12:24:47.075965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:47.075928    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce9c3b4-pnx8w"]
Oct 30 12:24:47.415541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:47.378012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:47.646824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:47.645520    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:47.801151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:47.801114    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:48.005475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.005022    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" event={"ID":"9352f0ab-4592-4083-a373-eb5780165613","Type":"ContainerStarted","Data":"1b42b5f6e698b2f3a4c0ecfc4f513fa8fbbe1099280f22d79b5f82f4246931f4"}
Oct 30 12:24:48.005475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.005074    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" event={"ID":"9352f0ab-4592-4083-a373-eb5780165613","Type":"ContainerStarted","Data":"dcb4bb72002fc9c94d7b15cc3cd1af6fd48299ba49a6c585ea42407b064b26f0"}
Oct 30 12:24:48.037799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.037768    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"]
Oct 30 12:24:48.042357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.042321    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fe7c53-96jbs"]
Oct 30 12:24:48.050732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.050687    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" podStartSLOduration=2.050673613 podStartE2EDuration="2.050673613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:48.048686233 +0000 UTC m=+3291.843570725" watchObservedRunningTime="2026-10-30 12:24:48.050673613 +0000 UTC m=+3291.845558106"
Oct 30 12:24:48.339027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.338990    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae53798-5b70-4ad1-a0b1-b8d6995fbb52" path="/var/lib/kubelet/pods/0ae53798-5b70-4ad1-a0b1-b8d6995fbb52/volumes"
Oct 30 12:24:48.339554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.339538    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eadd852-c051-4788-a118-9cc91bc2c1ce" path="/var/lib/kubelet/pods/0eadd852-c051-4788-a118-9cc91bc2c1ce/volumes"
Oct 30 12:24:48.572371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:48.572341    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:49.760713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:49.760672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:49.871896 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:24:49.871860    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:24:50.627354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.627313    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"]
Oct 30 12:24:50.634232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.634195    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.659492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.659169    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"]
Oct 30 12:24:50.660129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.660102    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5cf206\"/\"kube-root-ca.crt\""
Oct 30 12:24:50.660307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.660121    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5cf206\"/\"openshift-service-ca.crt\""
Oct 30 12:24:50.665509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.665433    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5cf206\"/\"default-dockercfg-dgrb8\""
Oct 30 12:24:50.665628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.665553    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5cf206\"/\"test-model-5cf206-kserve-self-signed-certs\""
Oct 30 12:24:50.720308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.720272    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:50.759838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.759781    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbz8\" (UniqueName: \"kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.760066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.759994    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.760066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.760042    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.760200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.760126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.760200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.760180    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.760313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.760287    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.862215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.862820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862333    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbz8\" (UniqueName: \"kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.862820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862444    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.862820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862509    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.862820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.862820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862590    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.863148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.862991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.863148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.863046    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.863148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.863070    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.864951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.864924    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.865225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.865207    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.895988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.889148    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbz8\" (UniqueName: \"kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8\") pod \"test-model-5cf206-kserve-dcfd96f99-gc4wn\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") " pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:50.972480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:50.962765    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:51.440840 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:24:51.440801    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd770124_119f_4a0c_8940_f4bff2a64ecf.slice/crio-fd3aa756cd7b962da8c9a1162a431b3f13fae14702f4e4a2989b3c2f96f61e87 WatchSource:0}: Error finding container fd3aa756cd7b962da8c9a1162a431b3f13fae14702f4e4a2989b3c2f96f61e87: Status 404 returned error can't find the container with id fd3aa756cd7b962da8c9a1162a431b3f13fae14702f4e4a2989b3c2f96f61e87
Oct 30 12:24:51.443974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.443939    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"]
Oct 30 12:24:51.457655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.457622    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"]
Oct 30 12:24:51.465108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.465084    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.497713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.497675    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"]
Oct 30 12:24:51.575926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.575900    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.575938    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.575966    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.576121    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.576143    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mcr\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.576289    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.576363    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.576422    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.576611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.576589    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.607358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.607323    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:51.678637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.678596    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.678863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.678709    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.678863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.678858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.678937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.678972    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679011    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679106    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679131    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mcr\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679209    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679344    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679391    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.679671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.679624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.682414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.682249    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.682607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.682528    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.682794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.682773    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.704748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.704724    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mcr\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.704911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.704762    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.765462 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.765407    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:51.799502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.799425    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:51.935505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:51.919353    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:52.208063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.208013    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" event={"ID":"fd770124-119f-4a0c-8940-f4bff2a64ecf","Type":"ContainerStarted","Data":"df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e"}
Oct 30 12:24:52.208277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.208069    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" event={"ID":"fd770124-119f-4a0c-8940-f4bff2a64ecf","Type":"ContainerStarted","Data":"fd3aa756cd7b962da8c9a1162a431b3f13fae14702f4e4a2989b3c2f96f61e87"}
Oct 30 12:24:52.253207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.253149    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" podStartSLOduration=2.253137354 podStartE2EDuration="2.253137354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:52.251444384 +0000 UTC m=+3296.046328879" watchObservedRunningTime="2026-10-30 12:24:52.253137354 +0000 UTC m=+3296.048021847"
Oct 30 12:24:52.328699 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:24:52.328670    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1933a8_46fa_420f_9b35_915792c4ea8e.slice/crio-691ad914cddd5f04ef5895324ac906659af396f3e3bf3723ec72f0b5244740a2 WatchSource:0}: Error finding container 691ad914cddd5f04ef5895324ac906659af396f3e3bf3723ec72f0b5244740a2: Status 404 returned error can't find the container with id 691ad914cddd5f04ef5895324ac906659af396f3e3bf3723ec72f0b5244740a2
Oct 30 12:24:52.332010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.331978    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:52.332117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.332051    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:52.332117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.332082    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:24:52.337761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.337737    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"]
Oct 30 12:24:52.963599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:52.963555    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:24:53.228061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.227051    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" event={"ID":"cc1933a8-46fa-420f-9b35-915792c4ea8e","Type":"ContainerStarted","Data":"f270ab04cb8f0b22222a82184b8ed0464ee2d3d6902e5a657e019985d8145bb5"}
Oct 30 12:24:53.228061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.227164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" event={"ID":"cc1933a8-46fa-420f-9b35-915792c4ea8e","Type":"ContainerStarted","Data":"691ad914cddd5f04ef5895324ac906659af396f3e3bf3723ec72f0b5244740a2"}
Oct 30 12:24:53.317151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.317091    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podStartSLOduration=2.317075702 podStartE2EDuration="2.317075702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:53.316004377 +0000 UTC m=+3297.110888888" watchObservedRunningTime="2026-10-30 12:24:53.317075702 +0000 UTC m=+3297.111960196"
Oct 30 12:24:53.474800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.474759    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:53.637106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.637015    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"]
Oct 30 12:24:53.642024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.641996    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.647121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.647093    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:53.657359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.657334    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0ecae0-dockercfg-qnlpk\""
Oct 30 12:24:53.657651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.657623    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"]
Oct 30 12:24:53.665346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.665321    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.665570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.665550    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2vl\" (UniqueName: \"kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl\") pod \"payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.692953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.692909    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0ecae0\""
Oct 30 12:24:53.834968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.783728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.834968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.783794    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2vl\" (UniqueName: \"kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl\") pod \"payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.834968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.785252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.906918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.883366    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:53.947509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.914251    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:53.947509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.914360    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 30 12:24:53.947509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.914411    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 30 12:24:53.947509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.941363    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2vl\" (UniqueName: \"kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl\") pod \"payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:53.982219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:53.982182    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:54.051435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:54.044189    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:54.410506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:54.409967    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:54.525659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:54.525184    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"]
Oct 30 12:24:54.805528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:54.804381    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 30 12:24:54.805528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:54.804438    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 30 12:24:55.269219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:55.269185    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" event={"ID":"d0721f4c-5439-493b-8383-73ef1e0051db","Type":"ContainerStarted","Data":"44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b"}
Oct 30 12:24:55.269702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:55.269685    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:24:55.269817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:55.269804    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" event={"ID":"d0721f4c-5439-493b-8383-73ef1e0051db","Type":"ContainerStarted","Data":"66935490bfe112935ab618eb1e9809f0ee1d1f2ef60aaf0cc7b366fb25b63070"}
Oct 30 12:24:55.324128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:55.324059    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" podStartSLOduration=2.324042741 podStartE2EDuration="2.324042741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:55.32364572 +0000 UTC m=+3299.118530216" watchObservedRunningTime="2026-10-30 12:24:55.324042741 +0000 UTC m=+3299.118927235"
Oct 30 12:24:55.807834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:55.800869    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 30 12:24:55.807834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:55.800930    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 30 12:24:56.142541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:56.129631    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:56.572643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:56.572587    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:56.616328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:56.616290    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:56.802888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:56.802852    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 30 12:24:56.803192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:56.803161    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 30 12:24:57.327423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:57.327383    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:24:57.648155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:57.648076    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:24:57.658573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:57.658548    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:24:57.802623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:57.802588    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 30 12:24:57.802921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:57.802892    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 30 12:24:58.034568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:58.034535    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"]
Oct 30 12:24:58.039208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:58.039179    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa0d27c-xlfkf"]
Oct 30 12:24:58.141249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:58.141200    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:24:58.338743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:58.338648    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92844707-ea61-400a-8a3c-e82304b3d122" path="/var/lib/kubelet/pods/92844707-ea61-400a-8a3c-e82304b3d122/volumes"
Oct 30 12:24:58.804561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:58.804522    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 30 12:24:58.805191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:58.805158    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 30 12:24:59.807643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.807609    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:59.808647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.808626    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:59.809354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.809322    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:24:59.852479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.852429    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"]
Oct 30 12:24:59.863508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.863478    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:24:59.869051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.868084    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"]
Oct 30 12:24:59.890310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.890280    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9d71fa\"/\"openshift-service-ca.crt\""
Oct 30 12:24:59.894440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.894419    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9d71fa\"/\"kube-root-ca.crt\""
Oct 30 12:24:59.894777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.894759    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9d71fa\"/\"test-model-9d71fa-kserve-self-signed-certs\""
Oct 30 12:24:59.900708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.900660    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9d71fa\"/\"default-dockercfg-f8455\""
Oct 30 12:24:59.940744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.940708    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:00.000359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.999546    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88jc\" (UniqueName: \"kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.000359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.999717    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.000359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.999779    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.000359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.999912    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.000359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:24:59.999960    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.000359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.000117    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.087943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.087851    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:00.101871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.101829    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.101988    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88jc\" (UniqueName: \"kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102199    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102249    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102348    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.102961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.102942    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.105845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.105812    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.106090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.106067    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.127304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.127243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88jc\" (UniqueName: \"kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc\") pod \"test-model-9d71fa-kserve-774989c5ff-6dwtm\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") " pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.228477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.225763    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:00.746154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.739857    5060 scope.go:122] "RemoveContainer" containerID="bcd9f4d0b328092808655af7c51cb6fdc19c96e1ecff0163340f1e6e28b77223"
Oct 30 12:25:00.778479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.775197    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"]
Oct 30 12:25:00.968418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.967986    5060 scope.go:122] "RemoveContainer" containerID="8e73435d8b67b2b5f8b100b4949282c29ad8b6a4bc7fb418377385f90d1c857b"
Oct 30 12:25:00.998881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:00.998618    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:25:01.136705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:01.131832    5060 scope.go:122] "RemoveContainer" containerID="f3ef4ad8a351e0b458e088abf2ff4d59c219310c991de14b46ae5ab877192892"
Oct 30 12:25:01.199780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:01.196876    5060 scope.go:122] "RemoveContainer" containerID="8fd81a5779dab9d57750025bb484ef585308ac958c30576764897c46ab892f2f"
Oct 30 12:25:01.324482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:01.319847    5060 scope.go:122] "RemoveContainer" containerID="270bf31f4f4d5b8404d5218507968172e646fe417d6a40b70952de1356f1df75"
Oct 30 12:25:01.609675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:01.608973    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" event={"ID":"447ba016-16e1-4916-97b9-ae82475d2557","Type":"ContainerStarted","Data":"d386502bdd24e080dd9efd5f7cb8500300c22f6b79aeb5679a4c2e2623bf9eaf"}
Oct 30 12:25:01.651490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:01.650825    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:25:01.702482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:01.681801    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:25:02.244274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:02.244236    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:02.628534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:02.626947    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" event={"ID":"447ba016-16e1-4916-97b9-ae82475d2557","Type":"ContainerStarted","Data":"6a5ea18fe13aad135774e7b9610adf2ebd1abb3859984d4cee0c6b633de52a24"}
Oct 30 12:25:02.671473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:02.671393    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" podStartSLOduration=3.671376852 podStartE2EDuration="3.671376852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:02.669618798 +0000 UTC m=+3306.464503318" watchObservedRunningTime="2026-10-30 12:25:02.671376852 +0000 UTC m=+3306.466261346"
Oct 30 12:25:03.227278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:03.227230    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:03.989044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:03.989006    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:04.671356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.671320    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"]
Oct 30 12:25:04.677850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.677827    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.695420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.695384    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"]
Oct 30 12:25:04.699336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.699308    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:25:04.699532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.699413    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:25:04.699620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.699547    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:25:04.704991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.704962    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:25:04.792960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.792926    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.793167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.793005    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.793167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.793145    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dghg\" (UniqueName: \"kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.895044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.895004    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.895234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.895094    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.895234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.895146    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dghg\" (UniqueName: \"kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.895827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.895804    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.897760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.897738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.921192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.921159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dghg\" (UniqueName: \"kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg\") pod \"maas-api-revoke-sub-d8994e8b-j82k6\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:04.995626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:04.995590    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:05.304852 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:05.304811    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3582bc_1d47_4be4_8b9a_18d7347f667d.slice/crio-cd62ff4ded4a470ae3175cb81afc0e820ab253d0793c96ba944c0d6f237260b1 WatchSource:0}: Error finding container cd62ff4ded4a470ae3175cb81afc0e820ab253d0793c96ba944c0d6f237260b1: Status 404 returned error can't find the container with id cd62ff4ded4a470ae3175cb81afc0e820ab253d0793c96ba944c0d6f237260b1
Oct 30 12:25:05.307609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:05.307435    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"]
Oct 30 12:25:05.658672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:05.658578    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6" event={"ID":"bc3582bc-1d47-4be4-8b9a-18d7347f667d","Type":"ContainerStarted","Data":"cd62ff4ded4a470ae3175cb81afc0e820ab253d0793c96ba944c0d6f237260b1"}
Oct 30 12:25:05.749664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:05.749626    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:06.298276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:06.298251    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:25:06.672584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:06.672494    5060 generic.go:356] "Generic (PLEG): container finished" podID="bc3582bc-1d47-4be4-8b9a-18d7347f667d" containerID="3cca9714d2b2b597f3f312609ff222da6558992555a93b8a2c13723e147ed523" exitCode=0
Oct 30 12:25:06.672584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:06.672541    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6" event={"ID":"bc3582bc-1d47-4be4-8b9a-18d7347f667d","Type":"ContainerDied","Data":"3cca9714d2b2b597f3f312609ff222da6558992555a93b8a2c13723e147ed523"}
Oct 30 12:25:07.541010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.540974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:07.873140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.873110    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:07.988007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.987972    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca\") pod \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") "
Oct 30 12:25:07.988191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.988063    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca\") pod \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") "
Oct 30 12:25:07.988191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.988125    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg\" (UniqueName: \"kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg\") pod \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\" (UID: \"bc3582bc-1d47-4be4-8b9a-18d7347f667d\") "
Oct 30 12:25:07.988377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.988347    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca" pod "bc3582bc-1d47-4be4-8b9a-18d7347f667d" (UID: "bc3582bc-1d47-4be4-8b9a-18d7347f667d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:07.988677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.988659    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:07.990443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.990418    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca" pod "bc3582bc-1d47-4be4-8b9a-18d7347f667d" (UID: "bc3582bc-1d47-4be4-8b9a-18d7347f667d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:07.990547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:07.990442    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg" pod "bc3582bc-1d47-4be4-8b9a-18d7347f667d" (UID: "bc3582bc-1d47-4be4-8b9a-18d7347f667d"). InnerVolumeSpecName "kube-api-access-2dghg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:08.090429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:08.090232    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3582bc-1d47-4be4-8b9a-18d7347f667d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:08.090429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:08.090271    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dghg\" (UniqueName: \"kubernetes.io/projected/bc3582bc-1d47-4be4-8b9a-18d7347f667d-kube-api-access-2dghg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:08.693674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:08.693593    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"
Oct 30 12:25:08.693674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:08.693619    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6" event={"ID":"bc3582bc-1d47-4be4-8b9a-18d7347f667d","Type":"ContainerDied","Data":"cd62ff4ded4a470ae3175cb81afc0e820ab253d0793c96ba944c0d6f237260b1"}
Oct 30 12:25:08.694094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:08.693677    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd62ff4ded4a470ae3175cb81afc0e820ab253d0793c96ba944c0d6f237260b1"
Oct 30 12:25:09.473652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:09.473541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:10.226867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:10.226833    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:10.240241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:10.240215    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:10.725130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:10.725101    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:25:11.155174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:11.155044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:12.034557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:12.034522    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"]
Oct 30 12:25:12.047068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:12.047036    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24dcb68d-vghtx"]
Oct 30 12:25:12.340303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:12.340225    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6cce878-2a2c-4105-946a-3b78a65531f6" path="/var/lib/kubelet/pods/a6cce878-2a2c-4105-946a-3b78a65531f6/volumes"
Oct 30 12:25:12.968125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:12.968089    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:14.751423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:14.751382    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:15.598497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.598444    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"]
Oct 30 12:25:15.599298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.599275    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bc3582bc-1d47-4be4-8b9a-18d7347f667d" containerName="revoke-keys"
Oct 30 12:25:15.599298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.599298    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3582bc-1d47-4be4-8b9a-18d7347f667d" containerName="revoke-keys"
Oct 30 12:25:15.599586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.599558    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3582bc-1d47-4be4-8b9a-18d7347f667d" containerName="revoke-keys"
Oct 30 12:25:15.604642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.604622    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.624632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.624592    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"]
Oct 30 12:25:15.627725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.627688    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:25:15.627851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.627762    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:25:15.627851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.627837    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:25:15.640506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.638911    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:25:15.721324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.721280    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.721570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.721501    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ftz\" (UniqueName: \"kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.721682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.721666    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.822767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.822723    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ftz\" (UniqueName: \"kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.823275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.822870    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.823275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.823007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.823622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.823595    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.825626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.825603    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.846255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.846227    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ftz\" (UniqueName: \"kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz\") pod \"maas-api-revoke-sub-a317845b-xmhc8\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:15.928560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:15.928435    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:16.240837 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:16.240800    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda249c5cb_3cf6_43d7_bd2d_e42882917ae1.slice/crio-7bfe370ce5d7f1890390ee4c256f79637d6428cd1545cc4e683b62c0c1e2223c WatchSource:0}: Error finding container 7bfe370ce5d7f1890390ee4c256f79637d6428cd1545cc4e683b62c0c1e2223c: Status 404 returned error can't find the container with id 7bfe370ce5d7f1890390ee4c256f79637d6428cd1545cc4e683b62c0c1e2223c
Oct 30 12:25:16.243045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:16.243016    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"]
Oct 30 12:25:16.458031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:16.457983    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:16.771224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:16.771131    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8" event={"ID":"a249c5cb-3cf6-43d7-bd2d-e42882917ae1","Type":"ContainerStarted","Data":"0b84dff20e3df3d709b9e373345a0819e65d36569ca9055251ebefffb68511ba"}
Oct 30 12:25:16.771224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:16.771181    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8" event={"ID":"a249c5cb-3cf6-43d7-bd2d-e42882917ae1","Type":"ContainerStarted","Data":"7bfe370ce5d7f1890390ee4c256f79637d6428cd1545cc4e683b62c0c1e2223c"}
Oct 30 12:25:16.815447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:16.815398    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8" podStartSLOduration=1.815384226 podStartE2EDuration="1.815384226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:16.813010576 +0000 UTC m=+3320.607895070" watchObservedRunningTime="2026-10-30 12:25:16.815384226 +0000 UTC m=+3320.610268719"
Oct 30 12:25:17.781220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:17.781179    5060 generic.go:356] "Generic (PLEG): container finished" podID="a249c5cb-3cf6-43d7-bd2d-e42882917ae1" containerID="0b84dff20e3df3d709b9e373345a0819e65d36569ca9055251ebefffb68511ba" exitCode=0
Oct 30 12:25:17.781694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:17.781234    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8" event={"ID":"a249c5cb-3cf6-43d7-bd2d-e42882917ae1","Type":"ContainerDied","Data":"0b84dff20e3df3d709b9e373345a0819e65d36569ca9055251ebefffb68511ba"}
Oct 30 12:25:18.216319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.216282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:18.982179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.982153    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:18.988417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.988394    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca\") pod \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") "
Oct 30 12:25:18.988609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.988537    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca\") pod \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") "
Oct 30 12:25:18.988688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.988614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz\" (UniqueName: \"kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz\") pod \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\" (UID: \"a249c5cb-3cf6-43d7-bd2d-e42882917ae1\") "
Oct 30 12:25:18.988820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.988797    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca" pod "a249c5cb-3cf6-43d7-bd2d-e42882917ae1" (UID: "a249c5cb-3cf6-43d7-bd2d-e42882917ae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:18.989248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.989224    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:18.990943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.990913    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca" pod "a249c5cb-3cf6-43d7-bd2d-e42882917ae1" (UID: "a249c5cb-3cf6-43d7-bd2d-e42882917ae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:18.990943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:18.990930    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz" pod "a249c5cb-3cf6-43d7-bd2d-e42882917ae1" (UID: "a249c5cb-3cf6-43d7-bd2d-e42882917ae1"). InnerVolumeSpecName "kube-api-access-28ftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:19.091330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:19.091247    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:19.091330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:19.091286    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ftz\" (UniqueName: \"kubernetes.io/projected/a249c5cb-3cf6-43d7-bd2d-e42882917ae1-kube-api-access-28ftz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:19.805669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:19.805620    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8" event={"ID":"a249c5cb-3cf6-43d7-bd2d-e42882917ae1","Type":"ContainerDied","Data":"7bfe370ce5d7f1890390ee4c256f79637d6428cd1545cc4e683b62c0c1e2223c"}
Oct 30 12:25:19.805669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:19.805673    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bfe370ce5d7f1890390ee4c256f79637d6428cd1545cc4e683b62c0c1e2223c"
Oct 30 12:25:19.805896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:19.805670    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"
Oct 30 12:25:19.859351 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:25:19.859313    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:25:20.093054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:20.092954    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:21.849714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:21.849678    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:23.665162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:23.665120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:25.506279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:25.506245    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:27.590485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:27.571915    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:27.770761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:27.741836    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:28.383534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.367755    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:25:28.404307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.404224    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:28.586829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.586784    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:25:28.721545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.695062    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"]
Oct 30 12:25:28.735515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.731177    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a249c5cb-3cf6-43d7-bd2d-e42882917ae1" containerName="revoke-keys"
Oct 30 12:25:28.735515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.731214    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a249c5cb-3cf6-43d7-bd2d-e42882917ae1" containerName="revoke-keys"
Oct 30 12:25:28.735515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:28.731490    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a249c5cb-3cf6-43d7-bd2d-e42882917ae1" containerName="revoke-keys"
Oct 30 12:25:29.220546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.220488    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"]
Oct 30 12:25:29.220965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.220946    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.418679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.416748    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:25:29.460000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.438966    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 12:25:29.460000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.445864    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": context deadline exceeded"
Oct 30 12:25:29.460000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.452666    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:25:29.460000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.452816    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:25:29.470508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.469220    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:25:29.470508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.469586    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 12:25:29.516317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.516207    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 12:25:29.517591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.517565    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:25:29.582562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.582515    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.583058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.583036    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdqg\" (UniqueName: \"kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.583208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.583193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.684671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.684636    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdqg\" (UniqueName: \"kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.684906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.684695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.684984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.684903    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.685668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.685609    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.687517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.687483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.708922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.708881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdqg\" (UniqueName: \"kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg\") pod \"maas-api-revoke-sub-6f9d52c0-dst5q\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:29.794143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:29.794034    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:30.113522 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:30.113490    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e328790_11da_4d08_b07b_66c9ab6b522b.slice/crio-4cc619e527d8fbda6de19d13aa2bff68fc3c4e87af79dde50eed134d2aeb4402 WatchSource:0}: Error finding container 4cc619e527d8fbda6de19d13aa2bff68fc3c4e87af79dde50eed134d2aeb4402: Status 404 returned error can't find the container with id 4cc619e527d8fbda6de19d13aa2bff68fc3c4e87af79dde50eed134d2aeb4402
Oct 30 12:25:30.116093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:30.116056    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"]
Oct 30 12:25:30.470439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:30.470394    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q" event={"ID":"8e328790-11da-4d08-b07b-66c9ab6b522b","Type":"ContainerStarted","Data":"4cc619e527d8fbda6de19d13aa2bff68fc3c4e87af79dde50eed134d2aeb4402"}
Oct 30 12:25:30.910386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:30.910287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:31.483726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:31.483689    5060 generic.go:356] "Generic (PLEG): container finished" podID="8e328790-11da-4d08-b07b-66c9ab6b522b" containerID="c0a5a7c7fd662eda909f1b1e731f586e0c7e8f1140ff4416359b53dd9afc8d34" exitCode=0
Oct 30 12:25:31.483928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:31.483767    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q" event={"ID":"8e328790-11da-4d08-b07b-66c9ab6b522b","Type":"ContainerDied","Data":"c0a5a7c7fd662eda909f1b1e731f586e0c7e8f1140ff4416359b53dd9afc8d34"}
Oct 30 12:25:31.994001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:31.993963    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"]
Oct 30 12:25:31.998831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:31.998808    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.021557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.021524    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"]
Oct 30 12:25:32.127302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.127247    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2bx\" (UniqueName: \"kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.127652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.127624    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.127818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.127801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.229908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.229871    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.230102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.230002    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2bx\" (UniqueName: \"kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.230102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.230098    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.230577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.230550    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.232697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.232670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.253939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.253871    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2bx\" (UniqueName: \"kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx\") pod \"maas-api-revoke-sub-a565d71d-tp9dp\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.316675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.316642    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:32.637337 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:32.636407    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2042a26_1cce_4243_af82_77c466a5a6bf.slice/crio-5441946b0685e2123f7ffd269ed8469e776d03fe7d716f7a5204fdebee028881 WatchSource:0}: Error finding container 5441946b0685e2123f7ffd269ed8469e776d03fe7d716f7a5204fdebee028881: Status 404 returned error can't find the container with id 5441946b0685e2123f7ffd269ed8469e776d03fe7d716f7a5204fdebee028881
Oct 30 12:25:32.640939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.640912    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"]
Oct 30 12:25:32.667097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.667071    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:32.766112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.766063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:32.839562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.839423    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca\") pod \"8e328790-11da-4d08-b07b-66c9ab6b522b\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") "
Oct 30 12:25:32.840286 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.840261    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca\") pod \"8e328790-11da-4d08-b07b-66c9ab6b522b\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") "
Oct 30 12:25:32.840628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.840608    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg\" (UniqueName: \"kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg\") pod \"8e328790-11da-4d08-b07b-66c9ab6b522b\" (UID: \"8e328790-11da-4d08-b07b-66c9ab6b522b\") "
Oct 30 12:25:32.840734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.840653    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca" pod "8e328790-11da-4d08-b07b-66c9ab6b522b" (UID: "8e328790-11da-4d08-b07b-66c9ab6b522b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:32.841339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.841312    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:32.878659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.878624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca" pod "8e328790-11da-4d08-b07b-66c9ab6b522b" (UID: "8e328790-11da-4d08-b07b-66c9ab6b522b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:32.878823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.878694    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg" pod "8e328790-11da-4d08-b07b-66c9ab6b522b" (UID: "8e328790-11da-4d08-b07b-66c9ab6b522b"). InnerVolumeSpecName "kube-api-access-hcdqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:32.942992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.942951    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdqg\" (UniqueName: \"kubernetes.io/projected/8e328790-11da-4d08-b07b-66c9ab6b522b-kube-api-access-hcdqg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:32.942992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:32.942988    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e328790-11da-4d08-b07b-66c9ab6b522b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:33.507480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:33.507398    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"
Oct 30 12:25:33.507956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:33.507438    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q" event={"ID":"8e328790-11da-4d08-b07b-66c9ab6b522b","Type":"ContainerDied","Data":"4cc619e527d8fbda6de19d13aa2bff68fc3c4e87af79dde50eed134d2aeb4402"}
Oct 30 12:25:33.507956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:33.507546    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cc619e527d8fbda6de19d13aa2bff68fc3c4e87af79dde50eed134d2aeb4402"
Oct 30 12:25:33.509836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:33.509758    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp" event={"ID":"f2042a26-1cce-4243-af82-77c466a5a6bf","Type":"ContainerStarted","Data":"4266152fbce078c800b7cd4e322cc77b0b0085ea3f8068256a794e5884ddd084"}
Oct 30 12:25:33.509836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:33.509832    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp" event={"ID":"f2042a26-1cce-4243-af82-77c466a5a6bf","Type":"ContainerStarted","Data":"5441946b0685e2123f7ffd269ed8469e776d03fe7d716f7a5204fdebee028881"}
Oct 30 12:25:33.554610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:33.554561    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp" podStartSLOduration=2.554543084 podStartE2EDuration="2.554543084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:33.5532795 +0000 UTC m=+3337.348163995" watchObservedRunningTime="2026-10-30 12:25:33.554543084 +0000 UTC m=+3337.349427579"
Oct 30 12:25:34.526640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:34.526602    5060 generic.go:356] "Generic (PLEG): container finished" podID="f2042a26-1cce-4243-af82-77c466a5a6bf" containerID="4266152fbce078c800b7cd4e322cc77b0b0085ea3f8068256a794e5884ddd084" exitCode=0
Oct 30 12:25:34.527082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:34.526699    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp" event={"ID":"f2042a26-1cce-4243-af82-77c466a5a6bf","Type":"ContainerDied","Data":"4266152fbce078c800b7cd4e322cc77b0b0085ea3f8068256a794e5884ddd084"}
Oct 30 12:25:34.621674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:34.621637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:35.734173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.734148    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:35.792356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.792262    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca\") pod \"f2042a26-1cce-4243-af82-77c466a5a6bf\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") "
Oct 30 12:25:35.792588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.792372    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx\" (UniqueName: \"kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx\") pod \"f2042a26-1cce-4243-af82-77c466a5a6bf\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") "
Oct 30 12:25:35.792588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.792552    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca\") pod \"f2042a26-1cce-4243-af82-77c466a5a6bf\" (UID: \"f2042a26-1cce-4243-af82-77c466a5a6bf\") "
Oct 30 12:25:35.792729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.792695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca" pod "f2042a26-1cce-4243-af82-77c466a5a6bf" (UID: "f2042a26-1cce-4243-af82-77c466a5a6bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:35.793217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.793194    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:35.795134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.795107    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca" pod "f2042a26-1cce-4243-af82-77c466a5a6bf" (UID: "f2042a26-1cce-4243-af82-77c466a5a6bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:35.795430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.795402    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx" pod "f2042a26-1cce-4243-af82-77c466a5a6bf" (UID: "f2042a26-1cce-4243-af82-77c466a5a6bf"). InnerVolumeSpecName "kube-api-access-sz2bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:35.894652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.894616    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2bx\" (UniqueName: \"kubernetes.io/projected/f2042a26-1cce-4243-af82-77c466a5a6bf-kube-api-access-sz2bx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:35.894652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:35.894646    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042a26-1cce-4243-af82-77c466a5a6bf-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:36.367081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:36.367044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:36.547067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:36.547032    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"
Oct 30 12:25:36.547067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:36.547061    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp" event={"ID":"f2042a26-1cce-4243-af82-77c466a5a6bf","Type":"ContainerDied","Data":"5441946b0685e2123f7ffd269ed8469e776d03fe7d716f7a5204fdebee028881"}
Oct 30 12:25:36.547289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:36.547112    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5441946b0685e2123f7ffd269ed8469e776d03fe7d716f7a5204fdebee028881"
Oct 30 12:25:38.162914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:38.162878    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:39.116163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.116117    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"]
Oct 30 12:25:39.117027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.117007    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8e328790-11da-4d08-b07b-66c9ab6b522b" containerName="revoke-keys"
Oct 30 12:25:39.117027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.117026    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e328790-11da-4d08-b07b-66c9ab6b522b" containerName="revoke-keys"
Oct 30 12:25:39.117165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.117065    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2042a26-1cce-4243-af82-77c466a5a6bf" containerName="revoke-keys"
Oct 30 12:25:39.117165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.117071    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2042a26-1cce-4243-af82-77c466a5a6bf" containerName="revoke-keys"
Oct 30 12:25:39.117234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.117199    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e328790-11da-4d08-b07b-66c9ab6b522b" containerName="revoke-keys"
Oct 30 12:25:39.117234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.117215    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2042a26-1cce-4243-af82-77c466a5a6bf" containerName="revoke-keys"
Oct 30 12:25:39.121214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.121196    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.138284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.138256    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"]
Oct 30 12:25:39.144655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.144588    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:25:39.144821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.144677    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:25:39.144821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.144785    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:25:39.148680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.148656    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:25:39.158373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.158343    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.158657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.158632    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.158782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.158688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccj9g\" (UniqueName: \"kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.259797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.259755    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.260204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.260014    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.260204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.260069    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccj9g\" (UniqueName: \"kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.260440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.260419    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.262704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.262679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.287825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.287792    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccj9g\" (UniqueName: \"kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g\") pod \"maas-api-revoke-sub-75e17a68-d2fn4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.437818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.437735    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:39.769213 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:39.769181    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1881e8f4_396a_447e_b4f6_b274a82910f4.slice/crio-260609b896b9421b6f4eb47f0ca7e6673f91fab9b7317207e682e231eae88a8c WatchSource:0}: Error finding container 260609b896b9421b6f4eb47f0ca7e6673f91fab9b7317207e682e231eae88a8c: Status 404 returned error can't find the container with id 260609b896b9421b6f4eb47f0ca7e6673f91fab9b7317207e682e231eae88a8c
Oct 30 12:25:39.771658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:39.771633    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"]
Oct 30 12:25:40.064371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:40.064274    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:40.589888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:40.589837    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4" event={"ID":"1881e8f4-396a-447e-b4f6-b274a82910f4","Type":"ContainerStarted","Data":"7678d5dc1fde6ccf023de5ee626151cf8550a0dffc48e5f3f8df8f2408766d2b"}
Oct 30 12:25:40.590356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:40.589899    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4" event={"ID":"1881e8f4-396a-447e-b4f6-b274a82910f4","Type":"ContainerStarted","Data":"260609b896b9421b6f4eb47f0ca7e6673f91fab9b7317207e682e231eae88a8c"}
Oct 30 12:25:40.631564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:40.631501    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4" podStartSLOduration=1.631485662 podStartE2EDuration="1.631485662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:40.63026636 +0000 UTC m=+3344.425150855" watchObservedRunningTime="2026-10-30 12:25:40.631485662 +0000 UTC m=+3344.426370156"
Oct 30 12:25:41.618849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:41.618808    5060 generic.go:356] "Generic (PLEG): container finished" podID="1881e8f4-396a-447e-b4f6-b274a82910f4" containerID="7678d5dc1fde6ccf023de5ee626151cf8550a0dffc48e5f3f8df8f2408766d2b" exitCode=0
Oct 30 12:25:41.619280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:41.618889    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4" event={"ID":"1881e8f4-396a-447e-b4f6-b274a82910f4","Type":"ContainerDied","Data":"7678d5dc1fde6ccf023de5ee626151cf8550a0dffc48e5f3f8df8f2408766d2b"}
Oct 30 12:25:41.768158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:41.768123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:42.642604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.642576    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"]
Oct 30 12:25:42.647511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.647489    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.698948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.698904    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"]
Oct 30 12:25:42.729575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.729533    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.729826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.729807    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbt77\" (UniqueName: \"kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.729958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.729945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.831680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.831649    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.831830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.831696    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbt77\" (UniqueName: \"kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.831830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.831745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.832676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.832520    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.834394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.834372    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.855650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.855626    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbt77\" (UniqueName: \"kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77\") pod \"maas-api-revoke-sub-3b97cd30-djscd\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:42.887653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.887585    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:42.933299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.933267    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca\") pod \"1881e8f4-396a-447e-b4f6-b274a82910f4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") "
Oct 30 12:25:42.933509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.933445    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca\") pod \"1881e8f4-396a-447e-b4f6-b274a82910f4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") "
Oct 30 12:25:42.933659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.933637    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca" pod "1881e8f4-396a-447e-b4f6-b274a82910f4" (UID: "1881e8f4-396a-447e-b4f6-b274a82910f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:42.933708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.933678    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g\" (UniqueName: \"kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g\") pod \"1881e8f4-396a-447e-b4f6-b274a82910f4\" (UID: \"1881e8f4-396a-447e-b4f6-b274a82910f4\") "
Oct 30 12:25:42.934292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.934253    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:42.935789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.935760    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca" pod "1881e8f4-396a-447e-b4f6-b274a82910f4" (UID: "1881e8f4-396a-447e-b4f6-b274a82910f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:42.935994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.935972    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g" pod "1881e8f4-396a-447e-b4f6-b274a82910f4" (UID: "1881e8f4-396a-447e-b4f6-b274a82910f4"). InnerVolumeSpecName "kube-api-access-ccj9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:42.983789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:42.983749    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:43.035323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.035287    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccj9g\" (UniqueName: \"kubernetes.io/projected/1881e8f4-396a-447e-b4f6-b274a82910f4-kube-api-access-ccj9g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:43.035323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.035321    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1881e8f4-396a-447e-b4f6-b274a82910f4-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:43.296509 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:43.296401    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afb84a8_3dc9_4e54_b8bd_ae1c9b065ce9.slice/crio-4494f6d907c76eacee5e167b642d0ff9ec191ee9ecb342e6c17f7846ff76ba05 WatchSource:0}: Error finding container 4494f6d907c76eacee5e167b642d0ff9ec191ee9ecb342e6c17f7846ff76ba05: Status 404 returned error can't find the container with id 4494f6d907c76eacee5e167b642d0ff9ec191ee9ecb342e6c17f7846ff76ba05
Oct 30 12:25:43.299169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.299145    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"]
Oct 30 12:25:43.538317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.537780    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:43.641155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.641064    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd" event={"ID":"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9","Type":"ContainerStarted","Data":"4494f6d907c76eacee5e167b642d0ff9ec191ee9ecb342e6c17f7846ff76ba05"}
Oct 30 12:25:43.642648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.642617    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"
Oct 30 12:25:43.643008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.642614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4" event={"ID":"1881e8f4-396a-447e-b4f6-b274a82910f4","Type":"ContainerDied","Data":"260609b896b9421b6f4eb47f0ca7e6673f91fab9b7317207e682e231eae88a8c"}
Oct 30 12:25:43.643008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:43.642740    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="260609b896b9421b6f4eb47f0ca7e6673f91fab9b7317207e682e231eae88a8c"
Oct 30 12:25:44.653825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:44.653731    5060 generic.go:356] "Generic (PLEG): container finished" podID="8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" containerID="aa82eb9b92d7d1dae60712d0d57a2516c4f770ee34478af65c282e38028d8b52" exitCode=0
Oct 30 12:25:44.654218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:44.653818    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd" event={"ID":"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9","Type":"ContainerDied","Data":"aa82eb9b92d7d1dae60712d0d57a2516c4f770ee34478af65c282e38028d8b52"}
Oct 30 12:25:45.032967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.032921    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"]
Oct 30 12:25:45.037711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.037682    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a9228d-l2fr9"]
Oct 30 12:25:45.315422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.315333    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:45.890192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.890156    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:45.995371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.995330    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca\") pod \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") "
Oct 30 12:25:45.995622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.995388    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77\" (UniqueName: \"kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77\") pod \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") "
Oct 30 12:25:45.995713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.995652    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca\") pod \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\" (UID: \"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9\") "
Oct 30 12:25:45.995825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.995804    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca" pod "8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" (UID: "8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:45.996551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.996524    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:45.998141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.998107    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77" pod "8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" (UID: "8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9"). InnerVolumeSpecName "kube-api-access-qbt77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:45.998403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:45.998383    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca" pod "8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" (UID: "8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:46.098014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:46.097981    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbt77\" (UniqueName: \"kubernetes.io/projected/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-kube-api-access-qbt77\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:46.098219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:46.098079    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:46.342026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:46.341936    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a51b58-6441-4658-b967-be80a9d581da" path="/var/lib/kubelet/pods/e3a51b58-6441-4658-b967-be80a9d581da/volumes"
Oct 30 12:25:46.676916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:46.676830    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"
Oct 30 12:25:46.676916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:46.676860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd" event={"ID":"8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9","Type":"ContainerDied","Data":"4494f6d907c76eacee5e167b642d0ff9ec191ee9ecb342e6c17f7846ff76ba05"}
Oct 30 12:25:46.676916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:46.676911    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4494f6d907c76eacee5e167b642d0ff9ec191ee9ecb342e6c17f7846ff76ba05"
Oct 30 12:25:47.008224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:47.008184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:47.163496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:47.161652    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:48.939957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:48.939916    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:49.858902 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:25:49.858865    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:25:50.751342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:50.751287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:50.900757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:50.900721    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:51.408066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:51.408030    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:51.730694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:51.730652    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:52.661489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:52.661414    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:54.001329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:54.001292    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:54.442430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:54.442330    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:54.598017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:54.597979    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:55.029506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.029463    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"]
Oct 30 12:25:55.029948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.029655    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj" podUID="9aef7f68-0ea1-4401-bdf3-4206d5244009" containerName="sim" containerID="cri-o://9fe13b942c6d5c36c09771e24202c3ee7d67b97ba8a76793db685fedef93e062" gracePeriod=30
Oct 30 12:25:55.376542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.375081    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:55.482995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.482952    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"]
Oct 30 12:25:55.484447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.484401    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1881e8f4-396a-447e-b4f6-b274a82910f4" containerName="revoke-keys"
Oct 30 12:25:55.484673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.484656    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1881e8f4-396a-447e-b4f6-b274a82910f4" containerName="revoke-keys"
Oct 30 12:25:55.484850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.484838    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" containerName="revoke-keys"
Oct 30 12:25:55.484948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.484936    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" containerName="revoke-keys"
Oct 30 12:25:55.485794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.485606    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" containerName="revoke-keys"
Oct 30 12:25:55.485955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.485943    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1881e8f4-396a-447e-b4f6-b274a82910f4" containerName="revoke-keys"
Oct 30 12:25:55.751333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.751304    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:55.772310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.765043    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"]
Oct 30 12:25:55.786332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.785436    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:25:55.828596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.828273    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:25:55.871084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.871055    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:25:55.871780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.817442    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:25:55.877251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.877222    5060 generic.go:356] "Generic (PLEG): container finished" podID="9aef7f68-0ea1-4401-bdf3-4206d5244009" containerID="9fe13b942c6d5c36c09771e24202c3ee7d67b97ba8a76793db685fedef93e062" exitCode=0
Oct 30 12:25:55.877731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.877705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj" event={"ID":"9aef7f68-0ea1-4401-bdf3-4206d5244009","Type":"ContainerDied","Data":"9fe13b942c6d5c36c09771e24202c3ee7d67b97ba8a76793db685fedef93e062"}
Oct 30 12:25:55.972993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.972958    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlht\" (UniqueName: \"kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:55.973179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.973159    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:55.973350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:55.973331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.053245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.053214    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:25:56.074163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.074132    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw\" (UniqueName: \"kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw\") pod \"9aef7f68-0ea1-4401-bdf3-4206d5244009\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") "
Oct 30 12:25:56.074426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.074413    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls\" (UniqueName: \"kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls\") pod \"9aef7f68-0ea1-4401-bdf3-4206d5244009\" (UID: \"9aef7f68-0ea1-4401-bdf3-4206d5244009\") "
Oct 30 12:25:56.074675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.074650    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.074790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.074772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.074956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.074936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlht\" (UniqueName: \"kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.075475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.075436    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.077066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.077034    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls" pod "9aef7f68-0ea1-4401-bdf3-4206d5244009" (UID: "9aef7f68-0ea1-4401-bdf3-4206d5244009"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:56.077208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.077176    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw" pod "9aef7f68-0ea1-4401-bdf3-4206d5244009" (UID: "9aef7f68-0ea1-4401-bdf3-4206d5244009"). InnerVolumeSpecName "kube-api-access-f9rxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:56.077280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.077232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.103591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.103557    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlht\" (UniqueName: \"kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht\") pod \"maas-api-revoke-sub-b3286f98-slfp7\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.146256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.146226    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:56.176405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.176371    5060 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9aef7f68-0ea1-4401-bdf3-4206d5244009-tls\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.176405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.176406    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rxw\" (UniqueName: \"kubernetes.io/projected/9aef7f68-0ea1-4401-bdf3-4206d5244009-kube-api-access-f9rxw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.463688 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:56.463658    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c30fb4b_09fc_4d92_862b_076d098ac88d.slice/crio-cd17ebc63b2a44c3b57a5de1e9b20fc1e6feae3f83905febc4acea642e9ef902 WatchSource:0}: Error finding container cd17ebc63b2a44c3b57a5de1e9b20fc1e6feae3f83905febc4acea642e9ef902: Status 404 returned error can't find the container with id cd17ebc63b2a44c3b57a5de1e9b20fc1e6feae3f83905febc4acea642e9ef902
Oct 30 12:25:56.466365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.466339    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"]
Oct 30 12:25:56.632596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.632442    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:56.892250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.892225    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"
Oct 30 12:25:56.892337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.892291    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj" event={"ID":"9aef7f68-0ea1-4401-bdf3-4206d5244009","Type":"ContainerDied","Data":"b77b790665f667c679b4f843a14eb3efbf0652b97a40ff883c6bf63d8569b32b"}
Oct 30 12:25:56.892396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.892344    5060 scope.go:122] "RemoveContainer" containerID="9fe13b942c6d5c36c09771e24202c3ee7d67b97ba8a76793db685fedef93e062"
Oct 30 12:25:56.896712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.896686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7" event={"ID":"1c30fb4b-09fc-4d92-862b-076d098ac88d","Type":"ContainerStarted","Data":"cd17ebc63b2a44c3b57a5de1e9b20fc1e6feae3f83905febc4acea642e9ef902"}
Oct 30 12:25:56.947281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.947248    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"]
Oct 30 12:25:56.956762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:56.956732    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-5cdfef/e2e-usage-sim-dc1a34-78b8b6fc9-qj4fj"]
Oct 30 12:25:57.859878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:57.859787    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:57.994503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:57.986754    5060 generic.go:356] "Generic (PLEG): container finished" podID="1c30fb4b-09fc-4d92-862b-076d098ac88d" containerID="fcd98497d835cf5cd190957060d7b9ccbb4b87f7ab06f59241049fda7d9263bf" exitCode=0
Oct 30 12:25:57.994503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:57.987145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7" event={"ID":"1c30fb4b-09fc-4d92-862b-076d098ac88d","Type":"ContainerDied","Data":"fcd98497d835cf5cd190957060d7b9ccbb4b87f7ab06f59241049fda7d9263bf"}
Oct 30 12:25:58.120538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.119578    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:58.339128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.339094    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aef7f68-0ea1-4401-bdf3-4206d5244009" path="/var/lib/kubelet/pods/9aef7f68-0ea1-4401-bdf3-4206d5244009/volumes"
Oct 30 12:25:58.345742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.345706    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:58.491993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.490421    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:58.720138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.720096    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"]
Oct 30 12:25:58.721420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.721393    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9aef7f68-0ea1-4401-bdf3-4206d5244009" containerName="sim"
Oct 30 12:25:58.721561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.721422    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aef7f68-0ea1-4401-bdf3-4206d5244009" containerName="sim"
Oct 30 12:25:58.721686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.721668    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aef7f68-0ea1-4401-bdf3-4206d5244009" containerName="sim"
Oct 30 12:25:58.729981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.729943    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.763067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.762990    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0190d8-openshift-default-dockercfg-wx46h\""
Oct 30 12:25:58.772155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.772122    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"]
Oct 30 12:25:58.816053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.815245    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:58.866964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.866907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867079    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867320    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867380    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867502    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhk6\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867686    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.867839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.867781    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.972765    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.972881    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.972921    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.972981    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.973037    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.973066    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.973141    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhk6\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.973299    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.973480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.973368    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.974167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.973921    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.977055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.974874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.977055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.975120    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.977055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.975345    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.988473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.985486    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:58.992475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:58.988713    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:59.022525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.013473    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:25:59.022525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.014437    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:59.028480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.023384    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:59.031539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.031144    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhk6\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6\") pod \"e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") " pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:59.071845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.071809    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:25:59.574895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.574863    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:25:59.637779 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:25:59.637740    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0720467_4a23_48da_b74f_10af54693218.slice/crio-17382168403ef481dba657ee6882430a654608f2f510f20509168103ee1a38c9 WatchSource:0}: Error finding container 17382168403ef481dba657ee6882430a654608f2f510f20509168103ee1a38c9: Status 404 returned error can't find the container with id 17382168403ef481dba657ee6882430a654608f2f510f20509168103ee1a38c9
Oct 30 12:25:59.641446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.641421    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"]
Oct 30 12:25:59.642873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.642840    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:25:59.642991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.642901    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:25:59.642991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.642930    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:25:59.710266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.708173    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht\" (UniqueName: \"kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht\") pod \"1c30fb4b-09fc-4d92-862b-076d098ac88d\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") "
Oct 30 12:25:59.710266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.709611    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca\") pod \"1c30fb4b-09fc-4d92-862b-076d098ac88d\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") "
Oct 30 12:25:59.710266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.709795    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca\") pod \"1c30fb4b-09fc-4d92-862b-076d098ac88d\" (UID: \"1c30fb4b-09fc-4d92-862b-076d098ac88d\") "
Oct 30 12:25:59.711698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.711654    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca" pod "1c30fb4b-09fc-4d92-862b-076d098ac88d" (UID: "1c30fb4b-09fc-4d92-862b-076d098ac88d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:59.715820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.715793    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca" pod "1c30fb4b-09fc-4d92-862b-076d098ac88d" (UID: "1c30fb4b-09fc-4d92-862b-076d098ac88d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:25:59.715972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.715886    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht" pod "1c30fb4b-09fc-4d92-862b-076d098ac88d" (UID: "1c30fb4b-09fc-4d92-862b-076d098ac88d"). InnerVolumeSpecName "kube-api-access-mwlht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:59.812441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.812383    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlht\" (UniqueName: \"kubernetes.io/projected/1c30fb4b-09fc-4d92-862b-076d098ac88d-kube-api-access-mwlht\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:59.812441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.812411    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:25:59.812441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:25:59.812422    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c30fb4b-09fc-4d92-862b-076d098ac88d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:26:00.237419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.237389    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"
Oct 30 12:26:00.238255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.238222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7" event={"ID":"1c30fb4b-09fc-4d92-862b-076d098ac88d","Type":"ContainerDied","Data":"cd17ebc63b2a44c3b57a5de1e9b20fc1e6feae3f83905febc4acea642e9ef902"}
Oct 30 12:26:00.238497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.238429    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd17ebc63b2a44c3b57a5de1e9b20fc1e6feae3f83905febc4acea642e9ef902"
Oct 30 12:26:00.239909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.239875    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" event={"ID":"c0720467-4a23-48da-b74f-10af54693218","Type":"ContainerStarted","Data":"e4b08865f0d0f8c69d1076eabc6f6a0ae1cb13a84a663efb2c6f883ffbfdc756"}
Oct 30 12:26:00.240032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.239927    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" event={"ID":"c0720467-4a23-48da-b74f-10af54693218","Type":"ContainerStarted","Data":"17382168403ef481dba657ee6882430a654608f2f510f20509168103ee1a38c9"}
Oct 30 12:26:00.299714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.299592    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podStartSLOduration=2.299555239 podStartE2EDuration="2.299555239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:00.285885359 +0000 UTC m=+3364.080769856" watchObservedRunningTime="2026-10-30 12:26:00.299555239 +0000 UTC m=+3364.094439734"
Oct 30 12:26:00.384282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:00.384244    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:01.074438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.074408    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:26:01.083232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.083189    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:26:01.249000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.248958    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:26:01.250112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.250090    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:26:01.888684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.888647    5060 scope.go:122] "RemoveContainer" containerID="74f2f62bfbce8c2243785b5326ddfb1edc306a5f5aeffa17605f6c1dc50a6981"
Oct 30 12:26:01.903802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.903777    5060 scope.go:122] "RemoveContainer" containerID="95001cac3ed4679616415505c5eda485e32364235a8fb515f54dc995688d1905"
Oct 30 12:26:01.915169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.915137    5060 scope.go:122] "RemoveContainer" containerID="210945bdd03526d5911a09ed503443971fa92b38768b48b54212d484f6fda5e5"
Oct 30 12:26:01.932211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:01.932187    5060 scope.go:122] "RemoveContainer" containerID="353f0db70f13244ffa87299e9ec1e837a2d3320edb06ac8091ec16244a56e858"
Oct 30 12:26:02.319261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:02.319216    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:02.733410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:02.733345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:02.895644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:02.895611    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:04.254849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:04.254806    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:04.434818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:04.434777    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:04.577080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:04.576980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:06.075104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.075067    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:06.205323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.205278    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"]
Oct 30 12:26:06.208895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.206802    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1c30fb4b-09fc-4d92-862b-076d098ac88d" containerName="revoke-keys"
Oct 30 12:26:06.208895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.206833    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c30fb4b-09fc-4d92-862b-076d098ac88d" containerName="revoke-keys"
Oct 30 12:26:06.208895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.207081    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c30fb4b-09fc-4d92-862b-076d098ac88d" containerName="revoke-keys"
Oct 30 12:26:06.214244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.214217    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.219526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.219492    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:06.222619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.222592    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"]
Oct 30 12:26:06.230231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.230203    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:26:06.242230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.242157    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:26:06.242230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.242188    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:26:06.242230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.242158    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:26:06.320100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.320062    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxl7h\" (UniqueName: \"kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.320339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.320310    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.320421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.320383    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.421480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.421369    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxl7h\" (UniqueName: \"kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.421675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.421594    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.421675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.421649    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.422410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.422386    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.424368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.424343    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.446255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.446224    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxl7h\" (UniqueName: \"kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h\") pod \"maas-api-revoke-sub-a3095ccd-48bp4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.535801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.535761    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:06.845789 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:26:06.845755    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e05969e_ada7_472a_a845_ecd9973dd9e4.slice/crio-c67945f4be51cb7d2664a01bbf5de90452d132245f2b5c54ba45bad0e975bae5 WatchSource:0}: Error finding container c67945f4be51cb7d2664a01bbf5de90452d132245f2b5c54ba45bad0e975bae5: Status 404 returned error can't find the container with id c67945f4be51cb7d2664a01bbf5de90452d132245f2b5c54ba45bad0e975bae5
Oct 30 12:26:06.847146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:06.847121    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"]
Oct 30 12:26:07.353325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:07.353233    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4" event={"ID":"9e05969e-ada7-472a-a845-ecd9973dd9e4","Type":"ContainerStarted","Data":"ce9ff964d2715cde1058522fbda2570c6a6ceb8ea8e095c7aece0f3c4ba9b63d"}
Oct 30 12:26:07.353325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:07.353291    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4" event={"ID":"9e05969e-ada7-472a-a845-ecd9973dd9e4","Type":"ContainerStarted","Data":"c67945f4be51cb7d2664a01bbf5de90452d132245f2b5c54ba45bad0e975bae5"}
Oct 30 12:26:07.395210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:07.395156    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4" podStartSLOduration=1.395140986 podStartE2EDuration="1.395140986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:07.39407747 +0000 UTC m=+3371.188961967" watchObservedRunningTime="2026-10-30 12:26:07.395140986 +0000 UTC m=+3371.190025480"
Oct 30 12:26:08.036765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:08.036725    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:08.364876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:08.364842    5060 generic.go:356] "Generic (PLEG): container finished" podID="9e05969e-ada7-472a-a845-ecd9973dd9e4" containerID="ce9ff964d2715cde1058522fbda2570c6a6ceb8ea8e095c7aece0f3c4ba9b63d" exitCode=0
Oct 30 12:26:08.365288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:08.364932    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4" event={"ID":"9e05969e-ada7-472a-a845-ecd9973dd9e4","Type":"ContainerDied","Data":"ce9ff964d2715cde1058522fbda2570c6a6ceb8ea8e095c7aece0f3c4ba9b63d"}
Oct 30 12:26:09.575058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.575026    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:09.714354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.704083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h\" (UniqueName: \"kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h\") pod \"9e05969e-ada7-472a-a845-ecd9973dd9e4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") "
Oct 30 12:26:09.714354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.704190    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca\") pod \"9e05969e-ada7-472a-a845-ecd9973dd9e4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") "
Oct 30 12:26:09.714354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.704343    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca\") pod \"9e05969e-ada7-472a-a845-ecd9973dd9e4\" (UID: \"9e05969e-ada7-472a-a845-ecd9973dd9e4\") "
Oct 30 12:26:09.714354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.710973    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca" pod "9e05969e-ada7-472a-a845-ecd9973dd9e4" (UID: "9e05969e-ada7-472a-a845-ecd9973dd9e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:09.732483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.722049    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h" pod "9e05969e-ada7-472a-a845-ecd9973dd9e4" (UID: "9e05969e-ada7-472a-a845-ecd9973dd9e4"). InnerVolumeSpecName "kube-api-access-hxl7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:09.732483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.723004    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca" pod "9e05969e-ada7-472a-a845-ecd9973dd9e4" (UID: "9e05969e-ada7-472a-a845-ecd9973dd9e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:26:09.814443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.802386    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:09.849815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.849728    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxl7h\" (UniqueName: \"kubernetes.io/projected/9e05969e-ada7-472a-a845-ecd9973dd9e4-kube-api-access-hxl7h\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:26:09.849815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.849765    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:26:09.849815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:09.849784    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e05969e-ada7-472a-a845-ecd9973dd9e4-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:26:10.431119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:10.429960    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4" event={"ID":"9e05969e-ada7-472a-a845-ecd9973dd9e4","Type":"ContainerDied","Data":"c67945f4be51cb7d2664a01bbf5de90452d132245f2b5c54ba45bad0e975bae5"}
Oct 30 12:26:10.431119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:10.430025    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c67945f4be51cb7d2664a01bbf5de90452d132245f2b5c54ba45bad0e975bae5"
Oct 30 12:26:10.431119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:10.430100    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"
Oct 30 12:26:11.648670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:11.648632    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:13.319280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:13.319231    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:15.068895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:15.068853    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:16.848150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:16.848115    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:18.187504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:18.187423    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:18.384825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:18.384710    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:19.072903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:19.072868    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:19.776849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:19.776808    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:20.015919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:20.015741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:20.121406 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:26:20.121313    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:26:20.157536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:20.157440    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:20.408162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:20.408071    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:20.896418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:20.896376    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:21.174651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:21.174552    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:22.979472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:22.979415    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:24.732617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:24.732577    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:26.410201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:26.410166    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:26.864006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:26.863972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:27.000824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:27.000785    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:27.372375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:27.372338    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:27.832574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:27.832534    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:28.020475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:28.019929    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:28.558183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:28.558140    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:30.324759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:30.317993    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:31.808245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:31.808207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:31.969081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:31.969027    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:32.177481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:32.177361    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:33.543564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:33.543526    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:33.687000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:33.686954    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:34.314927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:34.314884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:36.155134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:36.155077    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:38.273672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:38.273629    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:40.167021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:40.166980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:41.863917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:41.863870    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:43.632678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:43.632644    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:45.526435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:45.526393    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:47.260700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:47.260659    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:48.978756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:48.978712    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:49.382189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:49.382154    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:49.858765 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:26:49.858715    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:26:50.731107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:50.731035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:51.966383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.966343    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"]
Oct 30 12:26:51.967158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.967141    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9e05969e-ada7-472a-a845-ecd9973dd9e4" containerName="revoke-keys"
Oct 30 12:26:51.967222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.967160    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e05969e-ada7-472a-a845-ecd9973dd9e4" containerName="revoke-keys"
Oct 30 12:26:51.967338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.967326    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e05969e-ada7-472a-a845-ecd9973dd9e4" containerName="revoke-keys"
Oct 30 12:26:51.972502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.972431    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:51.990949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.990915    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"]
Oct 30 12:26:51.994968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.994943    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae3538\"/\"default-dockercfg-7fhkt\""
Oct 30 12:26:51.999860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:51.999832    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae3538\"/\"test-model-ae3538-kserve-self-signed-certs\""
Oct 30 12:26:52.000211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.000188    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae3538\"/\"kube-root-ca.crt\""
Oct 30 12:26:52.000349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.000204    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae3538\"/\"openshift-service-ca.crt\""
Oct 30 12:26:52.053223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.053185    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:52.096305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.096260    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.096573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.096472    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp78d\" (UniqueName: \"kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.096666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.096576    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.096666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.096607    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.096799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.096675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.096799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.096792    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.199175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.199128    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.199848    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200065    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200289    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200350    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200415    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp78d\" (UniqueName: \"kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200497    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200527    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.200941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.200915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.203887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.203861    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.204114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.204093    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.227503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.225914    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp78d\" (UniqueName: \"kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d\") pod \"test-model-ae3538-kserve-5b645f7d9-jlxbj\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") " pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.298103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.297776    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:52.440397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.440353    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:52.901580 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:26:52.885937    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc94dfe3c_1456_4ba2_9c76_8f07f9296871.slice/crio-900c337f24f2c92c2e523d78cb38407f02f3478637da6d34834bbad07854f076 WatchSource:0}: Error finding container 900c337f24f2c92c2e523d78cb38407f02f3478637da6d34834bbad07854f076: Status 404 returned error can't find the container with id 900c337f24f2c92c2e523d78cb38407f02f3478637da6d34834bbad07854f076
Oct 30 12:26:52.901580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:52.889621    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"]
Oct 30 12:26:53.135850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:53.135815    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" event={"ID":"c94dfe3c-1456-4ba2-9c76-8f07f9296871","Type":"ContainerStarted","Data":"1bf1ab6f5b875d63f5e15a31cef9610a7fa63a75670b54133f396d64b248ef4e"}
Oct 30 12:26:53.135850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:53.135860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" event={"ID":"c94dfe3c-1456-4ba2-9c76-8f07f9296871","Type":"ContainerStarted","Data":"900c337f24f2c92c2e523d78cb38407f02f3478637da6d34834bbad07854f076"}
Oct 30 12:26:53.181682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:53.181565    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" podStartSLOduration=2.181546518 podStartE2EDuration="2.181546518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:53.176572046 +0000 UTC m=+3416.971456538" watchObservedRunningTime="2026-10-30 12:26:53.181546518 +0000 UTC m=+3416.976431017"
Oct 30 12:26:53.299693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:53.299616    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:26:54.129829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:54.129786    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:54.279333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:54.279296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:54.706903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:54.706869    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:55.997980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:55.997946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:57.826065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:57.826028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:57.977373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:57.977338    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:58.202763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:58.202726    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:26:59.737964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:26:59.737924    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:01.434614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:01.434566    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:01.986570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:01.986536    5060 scope.go:122] "RemoveContainer" containerID="3ce2a97b858d187b2c67eb3db68f7b5392a6843f00d9eff1ff0b9a19fe23d221"
Oct 30 12:27:02.001301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:02.001279    5060 scope.go:122] "RemoveContainer" containerID="95ff4363c009b81f4f6eec0f43fee93110ec3dee6cdf0a5a7ca9894d59fa8277"
Oct 30 12:27:02.298722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:02.298635    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:27:02.313079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:02.313029    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:27:03.155671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:03.155637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:03.257309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:03.257269    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:27:04.032419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:04.032383    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"]
Oct 30 12:27:04.044397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:04.044364    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefeb1eb-vkv87"]
Oct 30 12:27:04.339428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:04.339350    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cda18ac-ae67-4bbd-be13-e386dbf09a34" path="/var/lib/kubelet/pods/4cda18ac-ae67-4bbd-be13-e386dbf09a34/volumes"
Oct 30 12:27:04.862630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:04.862596    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:06.559201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:06.559159    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:08.349761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:08.349716    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:10.091027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:10.090972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:11.955337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:11.955282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:13.650330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:13.650287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:15.033511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:15.033477    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"]
Oct 30 12:27:15.045097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:15.045069    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc017f-vhhqf"]
Oct 30 12:27:15.318645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:15.318534    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:16.352239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:16.352200    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1e49c9-026c-459e-bd26-98cd5ec3a974" path="/var/lib/kubelet/pods/bc1e49c9-026c-459e-bd26-98cd5ec3a974/volumes"
Oct 30 12:27:17.061571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.061524    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:17.531186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.531152    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"]
Oct 30 12:27:17.539196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.539167    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.554169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.554136    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"]
Oct 30 12:27:17.558112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.558074    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0ecae0\"/\"test-model-0ecae0-kserve-self-signed-certs\""
Oct 30 12:27:17.558593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.557971    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0ecae0\"/\"openshift-service-ca.crt\""
Oct 30 12:27:17.566558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.566534    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0ecae0\"/\"default-dockercfg-5hm74\""
Oct 30 12:27:17.566698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.566567    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0ecae0\"/\"kube-root-ca.crt\""
Oct 30 12:27:17.610203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.610149    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:17.722680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.722636    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.722965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.722946    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.723079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.723066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.723472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.723224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.723666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.723646    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7f5\" (UniqueName: \"kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.723792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.723779    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.829476    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.829552    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.829639    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.829692    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7f5\" (UniqueName: \"kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.829736    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.830060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.831895    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.832138    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.833539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.832368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.864472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.861857    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.888478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.878785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:17.911497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:17.893123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7f5\" (UniqueName: \"kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5\") pod \"test-model-0ecae0-kserve-689f596687-wmf4g\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") " pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:18.157637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:18.157562    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:18.469965 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:27:18.469935    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7eaa66b_7e75_4bb9_9fd1_f21e7f2e4d4b.slice/crio-aedf6d36e10897b258d221d343d51248937822966fa7a1077dc972647293d119 WatchSource:0}: Error finding container aedf6d36e10897b258d221d343d51248937822966fa7a1077dc972647293d119: Status 404 returned error can't find the container with id aedf6d36e10897b258d221d343d51248937822966fa7a1077dc972647293d119
Oct 30 12:27:18.472876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:18.472847    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"]
Oct 30 12:27:18.988156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:18.988122    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:19.441228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:19.441130    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g" event={"ID":"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b","Type":"ContainerStarted","Data":"87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e"}
Oct 30 12:27:19.441228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:19.441188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g" event={"ID":"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b","Type":"ContainerStarted","Data":"aedf6d36e10897b258d221d343d51248937822966fa7a1077dc972647293d119"}
Oct 30 12:27:19.491468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:19.491404    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g" podStartSLOduration=2.491389997 podStartE2EDuration="2.491389997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:19.486181678 +0000 UTC m=+3443.281066174" watchObservedRunningTime="2026-10-30 12:27:19.491389997 +0000 UTC m=+3443.286274490"
Oct 30 12:27:19.860568 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:27:19.860487    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:27:20.159784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:20.159677    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:21.239474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.234221    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 12:27:21.255473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.252606    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" podUID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" containerName="main" probeResult="failure" output="Get \"https://10.132.1.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.255473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.254270    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main" probeResult="failure" output="Get \"https://10.132.1.86:8000/health\": context deadline exceeded"
Oct 30 12:27:21.322604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.322551    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.516654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.516546    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.519180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.519144    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.535388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.535348    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.657697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.657654    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main" probeResult="failure" output="Get \"https://10.132.1.83:8000/ready\": context deadline exceeded"
Oct 30 12:27:21.698367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.698326    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:21.720360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.720307    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main" probeResult="failure" output="Get \"https://10.132.1.86:8000/ready\": context deadline exceeded"
Oct 30 12:27:21.799877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.799767    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.800546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.800117    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 12:27:21.810121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.810084    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.811962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.811934    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 12:27:21.871273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.871175    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" podUID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:21.968588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:21.968535    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main" probeResult="failure" output="Get \"https://10.132.1.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:27:23.475952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:23.475920    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:25.273993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:25.273954    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:27.063161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:27.063120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:28.158731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:28.158696    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:28.171529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:28.171502    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:28.751442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:28.751411    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:27:28.808080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:28.808047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:28.963671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:28.963584    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:30.711794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:30.711754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:32.560319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:32.560269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:34.303992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:34.303946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:36.052235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:36.052197    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:37.758165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:37.758121    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:39.450359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:39.450312    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:40.180669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.180634    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"]
Oct 30 12:27:40.186211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.186189    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.208699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.208669    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"]
Oct 30 12:27:40.220806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.220766    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5cf206-openshift-default-dockercfg-7fp89\""
Oct 30 12:27:40.288346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288311    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sfwl\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288354    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288425    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288527    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288552    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288668    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288705    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288770    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.288890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.288875    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.390896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.390857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.390950    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391084    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sfwl\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391125    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391143    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391160    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391243    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391298    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391395    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391751    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391853    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.391926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.392130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.391933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.393592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.393571    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.393734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.393717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.414969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.414932    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.415146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.415100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sfwl\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.502465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.502413    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:40.833062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.833032    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"]
Oct 30 12:27:40.834238 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:27:40.834209    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61258bd_cad3_4581_a458_1c89606335c5.slice/crio-34a355d2d2e0a6583ccd773caa17209f1a70a5b89ab0412dff2e8476450088a0 WatchSource:0}: Error finding container 34a355d2d2e0a6583ccd773caa17209f1a70a5b89ab0412dff2e8476450088a0: Status 404 returned error can't find the container with id 34a355d2d2e0a6583ccd773caa17209f1a70a5b89ab0412dff2e8476450088a0
Oct 30 12:27:40.837442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.837411    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:27:40.837564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.837513    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:27:40.837564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.837545    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:27:40.887306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:40.887275    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" event={"ID":"d61258bd-cad3-4581-a458-1c89606335c5","Type":"ContainerStarted","Data":"34a355d2d2e0a6583ccd773caa17209f1a70a5b89ab0412dff2e8476450088a0"}
Oct 30 12:27:41.315149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:41.315100    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:41.913578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:41.913545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" event={"ID":"d61258bd-cad3-4581-a458-1c89606335c5","Type":"ContainerStarted","Data":"00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de"}
Oct 30 12:27:41.976503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:41.976410    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podStartSLOduration=1.976395604 podStartE2EDuration="1.976395604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:41.971035218 +0000 UTC m=+3465.765919713" watchObservedRunningTime="2026-10-30 12:27:41.976395604 +0000 UTC m=+3465.771280097"
Oct 30 12:27:42.505481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:42.504802    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:42.507477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:42.507247    5060 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 12:27:42.507477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:42.507304    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 12:27:43.191478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:43.189028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:43.504478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:43.503915    5060 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 12:27:43.504478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:43.503986    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 12:27:44.503658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:44.503617    5060 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 12:27:44.504205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:44.503687    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 12:27:44.958984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:44.958950    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:45.506478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:45.505802    5060 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 12:27:45.506478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:45.505879    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 12:27:46.505480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:46.504264    5060 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 12:27:46.505480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:46.504325    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 12:27:46.637828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:46.637792    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:47.521602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:47.521571    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:47.522107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:47.522081    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:47.522894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:47.522871    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:27:48.380912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:48.380873    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:49.860440 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:27:49.860402    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:27:50.133164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:50.133076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:51.832790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:51.832755    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:53.261305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:53.261266    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:53.401540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:53.401468    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:53.665737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:53.665594    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:54.252741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:54.252707    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:55.402306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:55.402264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:57.166694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:57.166649    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:27:59.001116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:27:59.001072    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:00.434552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:00.434508    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:00.788481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:00.785508    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:00.931380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:00.931347    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:01.287876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.287837    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"]
Oct 30 12:28:01.297168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.297136    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.320442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.320412    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd0ff7-openshift-default-dockercfg-x8btz\""
Oct 30 12:28:01.355094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.355065    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"]
Oct 30 12:28:01.376475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.376367    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:01.434517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.430758    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.434803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.434769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.434893    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxqs\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.434959    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.435066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.435111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.435162    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.435205    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.435493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.435281    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.613589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.613112    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.613589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.613198    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.613589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.613234    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.613589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.613273    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.613589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.613307    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.613589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.613356    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.659485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.644948    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.659485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.645678    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.659485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.646234    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.719476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.670531    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.719476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.670615    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.719476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.670758    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxqs\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.719476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.671525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.719476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.671785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.719476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.694843    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:01.727420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.727385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxqs\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.751202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.750033    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.751202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.751096    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.753483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.751953    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo\") pod \"e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") " pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:01.988965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:01.988933    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:02.052036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:02.051142    5060 scope.go:122] "RemoveContainer" containerID="9e28012345e33d3ef7481316a5b622ae41439417e3a735a5802ff55c616f3a6e"
Oct 30 12:28:02.428255 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:28:02.428220    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a987869_2c8a_4ac3_85e1_2f3ebceaf85a.slice/crio-c9c7e6aac6800792e06e6cd36218c3dc233f89d9422b9ef240b0bbc7bc439f6b WatchSource:0}: Error finding container c9c7e6aac6800792e06e6cd36218c3dc233f89d9422b9ef240b0bbc7bc439f6b: Status 404 returned error can't find the container with id c9c7e6aac6800792e06e6cd36218c3dc233f89d9422b9ef240b0bbc7bc439f6b
Oct 30 12:28:02.428401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:02.428284    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"]
Oct 30 12:28:02.431677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:02.431646    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:02.432041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:02.432021    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:02.432174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:02.432161    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:02.870200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:02.870158    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:03.187617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:03.187530    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" event={"ID":"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a","Type":"ContainerStarted","Data":"ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"}
Oct 30 12:28:03.187785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:03.187640    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" event={"ID":"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a","Type":"ContainerStarted","Data":"c9c7e6aac6800792e06e6cd36218c3dc233f89d9422b9ef240b0bbc7bc439f6b"}
Oct 30 12:28:03.241024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:03.240962    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podStartSLOduration=2.240939801 podStartE2EDuration="2.240939801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:03.232830715 +0000 UTC m=+3487.027715221" watchObservedRunningTime="2026-10-30 12:28:03.240939801 +0000 UTC m=+3487.035824295"
Oct 30 12:28:03.990137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:03.990101    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:04.000663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:04.000632    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:04.198977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:04.198942    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:04.199975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:04.199954    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:28:04.491738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:04.491693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:05.725608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:05.725566    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:05.876409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:05.876369    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:06.583977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:06.583948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:07.236142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.236108    5060 generic.go:356] "Generic (PLEG): container finished" podID="c0720467-4a23-48da-b74f-10af54693218" containerID="e4b08865f0d0f8c69d1076eabc6f6a0ae1cb13a84a663efb2c6f883ffbfdc756" exitCode=0
Oct 30 12:28:07.236700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.236139    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" event={"ID":"c0720467-4a23-48da-b74f-10af54693218","Type":"ContainerDied","Data":"e4b08865f0d0f8c69d1076eabc6f6a0ae1cb13a84a663efb2c6f883ffbfdc756"}
Oct 30 12:28:07.236768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.236733    5060 scope.go:122] "RemoveContainer" containerID="e4b08865f0d0f8c69d1076eabc6f6a0ae1cb13a84a663efb2c6f883ffbfdc756"
Oct 30 12:28:07.239334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.239299    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:07.239497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.239388    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:07.239497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.239474    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:07.503488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:07.495032    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:08.062506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.062310    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"]
Oct 30 12:28:08.072760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.072730    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.095066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.095035    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dd0ff7\""
Oct 30 12:28:08.125479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.112352    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2tj\" (UniqueName: \"kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj\") pod \"payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.134641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.134609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.211368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.211336    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dd0ff7-dockercfg-l2zj6\""
Oct 30 12:28:08.231230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.222124    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"]
Oct 30 12:28:08.244834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.244781    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2tj\" (UniqueName: \"kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj\") pod \"payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.246284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.246256    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.248388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.248368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.270779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.270745    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" event={"ID":"c0720467-4a23-48da-b74f-10af54693218","Type":"ContainerStarted","Data":"aa6272ac63accd8b06c57b6c2b3a77663a9b10b66e70563ddb614950893b6eec"}
Oct 30 12:28:08.282742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.282701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2tj\" (UniqueName: \"kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj\") pod \"payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.511224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.511177    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:08.839262 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:28:08.839212    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b2bceb_8b85_4d64_a4ee_03a590df1fed.slice/crio-36178b80783ef98ee1cb7d3da87af9d11aabd3ff2cb1c581c0d1231fcdd63806 WatchSource:0}: Error finding container 36178b80783ef98ee1cb7d3da87af9d11aabd3ff2cb1c581c0d1231fcdd63806: Status 404 returned error can't find the container with id 36178b80783ef98ee1cb7d3da87af9d11aabd3ff2cb1c581c0d1231fcdd63806
Oct 30 12:28:08.841356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:08.841327    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"]
Oct 30 12:28:09.060193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.060143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:09.074083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.074052    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:28:09.075052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.075021    5060 patch_prober.go:29] interesting pod/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body=
Oct 30 12:28:09.075144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.075065    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused"
Oct 30 12:28:09.389477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.380960    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" event={"ID":"e8b2bceb-8b85-4d64-a4ee-03a590df1fed","Type":"ContainerStarted","Data":"c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed"}
Oct 30 12:28:09.389477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.381027    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" event={"ID":"e8b2bceb-8b85-4d64-a4ee-03a590df1fed","Type":"ContainerStarted","Data":"36178b80783ef98ee1cb7d3da87af9d11aabd3ff2cb1c581c0d1231fcdd63806"}
Oct 30 12:28:09.389477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.381083    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:09.540475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.539360    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" podStartSLOduration=2.539338566 podStartE2EDuration="2.539338566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:09.43063945 +0000 UTC m=+3493.225523945" watchObservedRunningTime="2026-10-30 12:28:09.539338566 +0000 UTC m=+3493.334223063"
Oct 30 12:28:09.543712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.543677    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"]
Oct 30 12:28:09.544153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:09.544119    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" containerID="cri-o://9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90" gracePeriod=30
Oct 30 12:28:10.086161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:10.086026    5060 patch_prober.go:29] interesting pod/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body=
Oct 30 12:28:10.086542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:10.086509    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused"
Oct 30 12:28:10.739782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:10.739739    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:11.073708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:11.073633    5060 patch_prober.go:29] interesting pod/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body=
Oct 30 12:28:11.073708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:11.073678    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused"
Oct 30 12:28:12.074486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:12.074429    5060 patch_prober.go:29] interesting pod/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body=
Oct 30 12:28:12.074911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:12.074526    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused"
Oct 30 12:28:12.508499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:12.507390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:12.814378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:12.814277    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:28:12.814378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:12.814338    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:28:13.073730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:13.073649    5060 patch_prober.go:29] interesting pod/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body=
Oct 30 12:28:13.073730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:13.073703    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused"
Oct 30 12:28:14.074489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.074433    5060 patch_prober.go:29] interesting pod/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body=
Oct 30 12:28:14.074882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.074502    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused"
Oct 30 12:28:14.266836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.266794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:14.877148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.877112    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:28:14.973281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973234    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973344    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973398    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973444    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973572    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973649    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973787    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:14.973927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973842    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.973927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:14.973927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973879    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:28:14.974166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.973935    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc\") pod \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\" (UID: \"f592eb9a-921f-4d16-91e6-63e9dd925b9e\") "
Oct 30 12:28:14.974166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.974052    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:14.974398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.974127    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:14.975002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.974976    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:14.975311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.975006    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:14.975311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.975022    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:14.975311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.975036    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:14.975311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.975052    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:14.979929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.979905    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:28:14.980031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.979920    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:14.980031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.979989    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:28:14.980031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:14.980024    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc" pod "f592eb9a-921f-4d16-91e6-63e9dd925b9e" (UID: "f592eb9a-921f-4d16-91e6-63e9dd925b9e"). InnerVolumeSpecName "kube-api-access-grkqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:28:15.076369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.076336    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkqc\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-kube-api-access-grkqc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:15.076369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.076366    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:15.076369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.076376    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:15.076874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.076386    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592eb9a-921f-4d16-91e6-63e9dd925b9e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:15.080375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.080347    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:28:15.080916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.080894    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:28:15.081844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.081793    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:28:15.450088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.450052    5060 generic.go:356] "Generic (PLEG): container finished" podID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerID="9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90" exitCode=0
Oct 30 12:28:15.450266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.450120    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"
Oct 30 12:28:15.450266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.450142    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" event={"ID":"f592eb9a-921f-4d16-91e6-63e9dd925b9e","Type":"ContainerDied","Data":"9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90"}
Oct 30 12:28:15.450266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.450211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn" event={"ID":"f592eb9a-921f-4d16-91e6-63e9dd925b9e","Type":"ContainerDied","Data":"ea727f04a00c1098926746584a3c015bd6d5caa09845c1fa8ce499a88bc591b1"}
Oct 30 12:28:15.450266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.450239    5060 scope.go:122] "RemoveContainer" containerID="9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90"
Oct 30 12:28:15.464083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.464063    5060 scope.go:122] "RemoveContainer" containerID="9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90"
Oct 30 12:28:15.464348 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:28:15.464331    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90\": container with ID starting with 9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90 not found: ID does not exist" containerID="9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90"
Oct 30 12:28:15.464414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.464357    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90"} err="failed to get container status \"9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90\": rpc error: code = NotFound desc = could not find container \"9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90\": container with ID starting with 9f85407a7a353202e5e6ff29e014b89d546d04a262554563627a037dbc3e5f90 not found: ID does not exist"
Oct 30 12:28:15.507781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.507751    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"]
Oct 30 12:28:15.513715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:15.513683    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-n4mfn"]
Oct 30 12:28:16.072051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:16.072019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:16.337883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:16.337806    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" path="/var/lib/kubelet/pods/f592eb9a-921f-4d16-91e6-63e9dd925b9e/volumes"
Oct 30 12:28:17.844906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:17.844870    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:19.527735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:19.527686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:19.859047 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:28:19.858955    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:28:21.219151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:21.219116    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"]
Oct 30 12:28:21.219740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:21.219385    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy" containerID="cri-o://c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4" gracePeriod=30
Oct 30 12:28:21.358210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:21.358176    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:21.429524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:21.429408    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:28:23.096301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:23.095704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:24.856571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:24.856525    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:26.522388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.522358    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:28:26.564734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.564702    5060 generic.go:356] "Generic (PLEG): container finished" podID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerID="c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4" exitCode=0
Oct 30 12:28:26.564906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.564743    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"
Oct 30 12:28:26.564906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.564786    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" event={"ID":"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc","Type":"ContainerDied","Data":"c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4"}
Oct 30 12:28:26.564906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.564837    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl" event={"ID":"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc","Type":"ContainerDied","Data":"56f1690536ee6a14821f2f3f5c417422cdcc589c8a41b1799c93f66150634709"}
Oct 30 12:28:26.564906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.564859    5060 scope.go:122] "RemoveContainer" containerID="c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4"
Oct 30 12:28:26.579801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.579698    5060 scope.go:122] "RemoveContainer" containerID="c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4"
Oct 30 12:28:26.580446 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:28:26.580373    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4\": container with ID starting with c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4 not found: ID does not exist" containerID="c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4"
Oct 30 12:28:26.580698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.580410    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4"} err="failed to get container status \"c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4\": rpc error: code = NotFound desc = could not find container \"c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4\": container with ID starting with c0c93051bb928f7582bc65bfdc08475f111ff5d90dd21b4d4ec4d38f2c7f8ac4 not found: ID does not exist"
Oct 30 12:28:26.596204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596177    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596335    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596386    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596503    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596544    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596646    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596761    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.596926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.596908    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket\") pod \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\" (UID: \"16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc\") "
Oct 30 12:28:26.598271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.598059    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:26.598271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.598241    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:26.598514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.598357    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:26.598746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.598715    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:28:26.601485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.601430    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "kube-api-access-wvtwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:28:26.606994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.606935    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:26.609135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.609100    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:26.609737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.609711    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:28:26.609842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.609795    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo" pod "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" (UID: "16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:28:26.652274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.652237    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:26.700360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700317    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700362    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700376    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700391    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700404    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700417    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvtwl\" (UniqueName: \"kubernetes.io/projected/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-kube-api-access-wvtwl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700431    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700443    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.700713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.700472    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:28:26.915260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.915224    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"]
Oct 30 12:28:26.927490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:26.927440    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-kjhjl"]
Oct 30 12:28:28.534261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:28.534224    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:28.549622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:28.542600    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" path="/var/lib/kubelet/pods/16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc/volumes"
Oct 30 12:28:29.804060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:29.804019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:30.316434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.316388    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:30.471404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.470896    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:30.726474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.725013    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"]
Oct 30 12:28:30.726474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.726234    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy"
Oct 30 12:28:30.726474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.726257    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy"
Oct 30 12:28:30.726474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.726313    5060 cpu_manager.go:414] "containerMap: removing container" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy"
Oct 30 12:28:30.726474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.726321    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy"
Oct 30 12:28:30.726903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.726598    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="16dcc4c7-8e41-4fe5-9bb5-32a5f9f86dcc" containerName="istio-proxy"
Oct 30 12:28:30.726903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.726633    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f592eb9a-921f-4d16-91e6-63e9dd925b9e" containerName="istio-proxy"
Oct 30 12:28:30.734810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.734782    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.755592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.755549    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"]
Oct 30 12:28:30.757542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.757431    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ffea51-openshift-default-dockercfg-t76tx\""
Oct 30 12:28:30.816609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.816568    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:30.828572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828536    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828728    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828760    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828797    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vh7\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828865    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828895    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.828980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.828939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.937541    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.937633    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.937744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.937810    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.937965    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.938018    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.938076    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.938116    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vh7\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.938176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.938686    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.939322    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.946475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.939929    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.976529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.961860    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.976529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.962112    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.976529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.970069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:30.976529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:30.975651    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:31.011479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.008872    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:31.020387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.012481    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:31.045347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.043828    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vh7\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7\") pod \"e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") " pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:31.074086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.072884    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:31.573082 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:28:31.573053    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bf4d13_9763_4407_9cd5_0fa4bbb628b1.slice/crio-29338aa355ffe5ba9cd6efc77984f8a11d9e6ac1a280d4892b73b9f972f76644 WatchSource:0}: Error finding container 29338aa355ffe5ba9cd6efc77984f8a11d9e6ac1a280d4892b73b9f972f76644: Status 404 returned error can't find the container with id 29338aa355ffe5ba9cd6efc77984f8a11d9e6ac1a280d4892b73b9f972f76644
Oct 30 12:28:31.576027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.576001    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"]
Oct 30 12:28:31.576391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.576361    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:31.576509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.576427    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:31.576596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.576540    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:28:31.641763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:31.641731    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" event={"ID":"68bf4d13-9763-4407-9cd5-0fa4bbb628b1","Type":"ContainerStarted","Data":"29338aa355ffe5ba9cd6efc77984f8a11d9e6ac1a280d4892b73b9f972f76644"}
Oct 30 12:28:32.257811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:32.257774    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:32.652840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:32.652754    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" event={"ID":"68bf4d13-9763-4407-9cd5-0fa4bbb628b1","Type":"ContainerStarted","Data":"b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"}
Oct 30 12:28:32.711863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:32.711802    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podStartSLOduration=2.711789875 podStartE2EDuration="2.711789875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:32.70933256 +0000 UTC m=+3516.504217055" watchObservedRunningTime="2026-10-30 12:28:32.711789875 +0000 UTC m=+3516.506674368"
Oct 30 12:28:33.076051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:33.076018    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:33.085091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:33.085059    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:33.661738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:33.661701    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:33.662603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:33.662584    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:28:34.155798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:34.155765    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:35.249259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:35.249218    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:35.402326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:35.402280    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:36.173255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:36.173210    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:37.003092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:37.003049    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:37.145800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:37.145764    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:38.212775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:38.212738    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:40.060052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:40.060017    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:41.746942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:41.746903    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:43.557349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:43.557310    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:45.259406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:45.259373    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:46.180002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:46.179961    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:47.070547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:47.070504    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:48.757336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:48.757302    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:49.859536 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:28:49.859497    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:28:50.436051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:50.436015    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:52.134278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:52.134224    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:53.861712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:53.861672    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:54.390193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:54.390160    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:55.596354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:55.596314    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:57.360604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:57.360559    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:28:59.095319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:28:59.095276    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:00.844544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:00.844511    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:02.148938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:02.148900    5060 scope.go:122] "RemoveContainer" containerID="bacc8cc73f387416dacf3f9da26fb9280d97355ee10c4313456b7c7a805bbd48"
Oct 30 12:29:02.702435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:02.702391    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:02.946131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:02.946094    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:03.087977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:03.087896    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"]
Oct 30 12:29:03.088146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:03.088125    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy" containerID="cri-o://9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9" gracePeriod=30
Oct 30 12:29:03.096417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:03.096339    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:04.741371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:04.741325    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:06.552799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:06.552711    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:08.226528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.225417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:08.452929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.452902    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:29:08.561808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.561774    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.561942    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.561971    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562030    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562103    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562112    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.562352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562272    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562324    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562533    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:29:08.562887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562590    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562642    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo\") pod \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\" (UID: \"64fdbf34-6a5f-4db3-983e-ba670dc24c27\") "
Oct 30 12:29:08.562887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562677    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.562887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562833    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.563270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.562895    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.563884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.563861    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.563993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.563888    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.563993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.563902    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.563993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.563920    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.563993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.563933    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.566410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.566386    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:08.566611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.566446    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "kube-api-access-72m5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:08.566749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.566725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.566994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.566973    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo" pod "64fdbf34-6a5f-4db3-983e-ba670dc24c27" (UID: "64fdbf34-6a5f-4db3-983e-ba670dc24c27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:29:08.665320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.665284    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72m5c\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-kube-api-access-72m5c\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.665320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.665319    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.665320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.665329    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.665626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:08.665337    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fdbf34-6a5f-4db3-983e-ba670dc24c27-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:09.072859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.072823    5060 generic.go:356] "Generic (PLEG): container finished" podID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerID="9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9" exitCode=0
Oct 30 12:29:09.073048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.072895    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"
Oct 30 12:29:09.073048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.072900    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" event={"ID":"64fdbf34-6a5f-4db3-983e-ba670dc24c27","Type":"ContainerDied","Data":"9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9"}
Oct 30 12:29:09.073048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.072993    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz" event={"ID":"64fdbf34-6a5f-4db3-983e-ba670dc24c27","Type":"ContainerDied","Data":"406a3b790ee2c0a3f490d6467e25e3a13f4727e91af023ebb90076f702be600f"}
Oct 30 12:29:09.073048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.073026    5060 scope.go:122] "RemoveContainer" containerID="9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9"
Oct 30 12:29:09.093735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.089908    5060 scope.go:122] "RemoveContainer" containerID="9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9"
Oct 30 12:29:09.093735 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:29:09.090347    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9\": container with ID starting with 9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9 not found: ID does not exist" containerID="9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9"
Oct 30 12:29:09.093735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.090389    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9"} err="failed to get container status \"9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9\": rpc error: code = NotFound desc = could not find container \"9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9\": container with ID starting with 9d1875d90547f983e660bec5c2ff8db4e1b0fd3725a7cf39e52612c6fec3c7a9 not found: ID does not exist"
Oct 30 12:29:09.142902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.142869    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"]
Oct 30 12:29:09.155256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.155221    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fp4jz"]
Oct 30 12:29:09.905328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:09.905291    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:10.338609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:10.338572    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" path="/var/lib/kubelet/pods/64fdbf34-6a5f-4db3-983e-ba670dc24c27/volumes"
Oct 30 12:29:11.665844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:11.665808    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:13.415212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:13.415123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:15.163394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:15.163339    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:16.886031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:16.885996    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:18.615209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:18.615168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:19.860540 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:29:19.860494    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:29:20.316829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:20.316781    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:22.058685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:22.058649    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:22.203400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:22.203363    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"]
Oct 30 12:29:22.203696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:22.203601    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy" containerID="cri-o://f270ab04cb8f0b22222a82184b8ed0464ee2d3d6902e5a657e019985d8145bb5" gracePeriod=30
Oct 30 12:29:23.850717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:23.850624    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:25.533870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:25.533828    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:27.256368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.256332    5060 generic.go:356] "Generic (PLEG): container finished" podID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerID="f270ab04cb8f0b22222a82184b8ed0464ee2d3d6902e5a657e019985d8145bb5" exitCode=0
Oct 30 12:29:27.256802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.256370    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" event={"ID":"cc1933a8-46fa-420f-9b35-915792c4ea8e","Type":"ContainerDied","Data":"f270ab04cb8f0b22222a82184b8ed0464ee2d3d6902e5a657e019985d8145bb5"}
Oct 30 12:29:27.334290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.334260    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"]
Oct 30 12:29:27.335081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.335067    5060 cpu_manager.go:414] "containerMap: removing container" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy"
Oct 30 12:29:27.335151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.335083    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy"
Oct 30 12:29:27.335261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.335250    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="64fdbf34-6a5f-4db3-983e-ba670dc24c27" containerName="istio-proxy"
Oct 30 12:29:27.339082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.339063    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.354735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.354709    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"]
Oct 30 12:29:27.363048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.361210    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:29:27.363048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.361248    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:29:27.363048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.361263    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:29:27.366401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.366376    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:29:27.371585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.371556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:27.404274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.403590    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5gh\" (UniqueName: \"kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.404274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.403743    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.404274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.403793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.505368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.505262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5gh\" (UniqueName: \"kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.505368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.505367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.505646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.505401    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.506133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.506109    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.508122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.508088    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.530020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.529989    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5gh\" (UniqueName: \"kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh\") pod \"maas-api-revoke-sub-77f4b1a3-bjwrn\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.655981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.655946    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:27.949069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.949043    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:29:27.972356 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:29:27.972330    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6581ae_7654_4848_82e3_dbac5e3ccec8.slice/crio-7523e87225889e1e4390e8cca5d6992b798bf730a159bf34475965ac2e8235de WatchSource:0}: Error finding container 7523e87225889e1e4390e8cca5d6992b798bf730a159bf34475965ac2e8235de: Status 404 returned error can't find the container with id 7523e87225889e1e4390e8cca5d6992b798bf730a159bf34475965ac2e8235de
Oct 30 12:29:27.982300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:27.982273    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"]
Oct 30 12:29:28.017257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.017212    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.017431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.017338    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.017578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.017435    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.017957    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018009    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018053    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018106    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018143    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018203    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018325    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data\") pod \"cc1933a8-46fa-420f-9b35-915792c4ea8e\" (UID: \"cc1933a8-46fa-420f-9b35-915792c4ea8e\") "
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018704    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018981    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.018998    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.019341    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.019942    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.019965    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1933a8-46fa-420f-9b35-915792c4ea8e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.019981    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.019994    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.020725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.020007    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.027978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.027948    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "kube-api-access-76mcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:28.028110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.027994    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:28.028110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.028017    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:29:28.028255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.028180    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy" pod "cc1933a8-46fa-420f-9b35-915792c4ea8e" (UID: "cc1933a8-46fa-420f-9b35-915792c4ea8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:28.121616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.121519    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76mcr\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-kube-api-access-76mcr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.121616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.121550    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.121616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.121561    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.121616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.121570    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1933a8-46fa-420f-9b35-915792c4ea8e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:28.267111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.267081    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"
Oct 30 12:29:28.267582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.267119    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74" event={"ID":"cc1933a8-46fa-420f-9b35-915792c4ea8e","Type":"ContainerDied","Data":"691ad914cddd5f04ef5895324ac906659af396f3e3bf3723ec72f0b5244740a2"}
Oct 30 12:29:28.267582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.267183    5060 scope.go:122] "RemoveContainer" containerID="f270ab04cb8f0b22222a82184b8ed0464ee2d3d6902e5a657e019985d8145bb5"
Oct 30 12:29:28.268705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.268682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn" event={"ID":"cc6581ae-7654-4848-82e3-dbac5e3ccec8","Type":"ContainerStarted","Data":"7523e87225889e1e4390e8cca5d6992b798bf730a159bf34475965ac2e8235de"}
Oct 30 12:29:28.326522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.326486    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"]
Oct 30 12:29:28.337908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:28.337879    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-ftx74"]
Oct 30 12:29:29.160497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:29.160352    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:29.282021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:29.281986    5060 generic.go:356] "Generic (PLEG): container finished" podID="cc6581ae-7654-4848-82e3-dbac5e3ccec8" containerID="bef0371641582458e891b8ee2af2fe86d676666e79deec385d03e72ba2363086" exitCode=0
Oct 30 12:29:29.282467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:29.282085    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn" event={"ID":"cc6581ae-7654-4848-82e3-dbac5e3ccec8","Type":"ContainerDied","Data":"bef0371641582458e891b8ee2af2fe86d676666e79deec385d03e72ba2363086"}
Oct 30 12:29:30.339630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.339586    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" path="/var/lib/kubelet/pods/cc1933a8-46fa-420f-9b35-915792c4ea8e/volumes"
Oct 30 12:29:30.494184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.494153    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:30.660750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.660668    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca\") pod \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") "
Oct 30 12:29:30.660899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.660761    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca\") pod \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") "
Oct 30 12:29:30.660899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.660839    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh\" (UniqueName: \"kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh\") pod \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\" (UID: \"cc6581ae-7654-4848-82e3-dbac5e3ccec8\") "
Oct 30 12:29:30.661159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.661130    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca" pod "cc6581ae-7654-4848-82e3-dbac5e3ccec8" (UID: "cc6581ae-7654-4848-82e3-dbac5e3ccec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:29:30.661594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.661575    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:30.663100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.663070    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca" pod "cc6581ae-7654-4848-82e3-dbac5e3ccec8" (UID: "cc6581ae-7654-4848-82e3-dbac5e3ccec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:29:30.663184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.663160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh" pod "cc6581ae-7654-4848-82e3-dbac5e3ccec8" (UID: "cc6581ae-7654-4848-82e3-dbac5e3ccec8"). InnerVolumeSpecName "kube-api-access-db5gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:30.762992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.762962    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5gh\" (UniqueName: \"kubernetes.io/projected/cc6581ae-7654-4848-82e3-dbac5e3ccec8-kube-api-access-db5gh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:30.762992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.762988    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6581ae-7654-4848-82e3-dbac5e3ccec8-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:30.849869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:30.849806    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:31.308474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:31.307439    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"
Oct 30 12:29:31.308474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:31.307497    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn" event={"ID":"cc6581ae-7654-4848-82e3-dbac5e3ccec8","Type":"ContainerDied","Data":"7523e87225889e1e4390e8cca5d6992b798bf730a159bf34475965ac2e8235de"}
Oct 30 12:29:31.308474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:31.307550    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7523e87225889e1e4390e8cca5d6992b798bf730a159bf34475965ac2e8235de"
Oct 30 12:29:32.630505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:32.630466    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:32.771004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:32.770936    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:34.545006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:34.544970    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:36.232521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:36.232482    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:38.065064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.065026    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:38.263567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.263521    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"]
Oct 30 12:29:38.332685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.332588    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:38.814659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.814625    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"]
Oct 30 12:29:38.815539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.815518    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cc6581ae-7654-4848-82e3-dbac5e3ccec8" containerName="revoke-keys"
Oct 30 12:29:38.815539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.815539    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6581ae-7654-4848-82e3-dbac5e3ccec8" containerName="revoke-keys"
Oct 30 12:29:38.815664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.815572    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy"
Oct 30 12:29:38.815664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.815578    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy"
Oct 30 12:29:38.815758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.815745    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6581ae-7654-4848-82e3-dbac5e3ccec8" containerName="revoke-keys"
Oct 30 12:29:38.815802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.815767    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1933a8-46fa-420f-9b35-915792c4ea8e" containerName="istio-proxy"
Oct 30 12:29:38.819944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.819928    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.834764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.834734    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:29:38.834957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.834856    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:29:38.836654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.836633    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"]
Oct 30 12:29:38.841891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.841863    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:29:38.842027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.841975    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:29:38.862709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.862667    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.862709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.862707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqff\" (UniqueName: \"kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.862977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.862739    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.964041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.964002    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.964041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.964043    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqff\" (UniqueName: \"kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.964317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.964069    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.964779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.964752    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.966650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.966634    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:38.987933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:38.987900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqff\" (UniqueName: \"kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff\") pod \"maas-api-revoke-sub-4afad348-vj4cz\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:39.137356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:39.137270    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:39.460418 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:29:39.460377    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a860a9_f5c9_4c22_b9d9_d90cc17c4be1.slice/crio-ab01fcdcb835b12cace5d7157c81031e052caa13a17bbd1793fd17d79c63cf9f WatchSource:0}: Error finding container ab01fcdcb835b12cace5d7157c81031e052caa13a17bbd1793fd17d79c63cf9f: Status 404 returned error can't find the container with id ab01fcdcb835b12cace5d7157c81031e052caa13a17bbd1793fd17d79c63cf9f
Oct 30 12:29:39.463077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:39.463052    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"]
Oct 30 12:29:39.541628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:39.541574    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz" event={"ID":"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1","Type":"ContainerStarted","Data":"ab01fcdcb835b12cace5d7157c81031e052caa13a17bbd1793fd17d79c63cf9f"}
Oct 30 12:29:39.843290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:39.843256    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:40.557546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:40.557499    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz" event={"ID":"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1","Type":"ContainerStarted","Data":"7bd8aaa51c2d42961273244d1f9212e78a5014916405ad2a50c5c99abc719df3"}
Oct 30 12:29:40.602750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:40.602188    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz" podStartSLOduration=2.602168904 podStartE2EDuration="2.602168904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:40.600476233 +0000 UTC m=+3584.395360723" watchObservedRunningTime="2026-10-30 12:29:40.602168904 +0000 UTC m=+3584.397053401"
Oct 30 12:29:41.199049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:41.199009    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:41.344518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:41.344481    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:41.670534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:41.665548    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:41.695475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:41.687440    5060 generic.go:356] "Generic (PLEG): container finished" podID="f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" containerID="7bd8aaa51c2d42961273244d1f9212e78a5014916405ad2a50c5c99abc719df3" exitCode=0
Oct 30 12:29:41.695475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:41.687650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz" event={"ID":"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1","Type":"ContainerDied","Data":"7bd8aaa51c2d42961273244d1f9212e78a5014916405ad2a50c5c99abc719df3"}
Oct 30 12:29:43.281088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.281054    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:43.361166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.361128    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff\" (UniqueName: \"kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff\") pod \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") "
Oct 30 12:29:43.361366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.361231    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca\") pod \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") "
Oct 30 12:29:43.361491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.361472    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca\") pod \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\" (UID: \"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1\") "
Oct 30 12:29:43.362721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.362691    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca" pod "f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" (UID: "f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:29:43.365598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.365560    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca" pod "f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" (UID: "f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:29:43.367367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.367338    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff" pod "f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" (UID: "f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1"). InnerVolumeSpecName "kube-api-access-5fqff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:43.464437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.464393    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fqff\" (UniqueName: \"kubernetes.io/projected/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-kube-api-access-5fqff\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:43.464437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.464435    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:43.464798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.464465    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:43.660307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.660198    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:43.805539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.799018    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz" event={"ID":"f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1","Type":"ContainerDied","Data":"ab01fcdcb835b12cace5d7157c81031e052caa13a17bbd1793fd17d79c63cf9f"}
Oct 30 12:29:43.823630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.805836    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"
Oct 30 12:29:43.824009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:43.823991    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab01fcdcb835b12cace5d7157c81031e052caa13a17bbd1793fd17d79c63cf9f"
Oct 30 12:29:45.549519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:45.549446    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:47.291076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:47.291029    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:47.882606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:47.882565    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:49.059231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:49.059184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:49.862133 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:29:49.862094    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:29:50.852291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:50.852213    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:52.622406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:52.622362    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:53.296090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.296043    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" podUID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" containerName="main" containerID="cri-o://4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab" gracePeriod=45
Oct 30 12:29:53.590670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.590640    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:29:53.677366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.677327    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs\") pod \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") "
Oct 30 12:29:53.677981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.677498    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd\" (UniqueName: \"kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd\") pod \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") "
Oct 30 12:29:53.677981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.677598    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home\") pod \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") "
Oct 30 12:29:53.677981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.677645    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache\") pod \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") "
Oct 30 12:29:53.677981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.677674    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir\") pod \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") "
Oct 30 12:29:53.677981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.677895    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm\") pod \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\" (UID: \"1f86dd20-d255-4a9e-baa2-5dd9521bb79c\") "
Oct 30 12:29:53.678292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.678017    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home" pod "1f86dd20-d255-4a9e-baa2-5dd9521bb79c" (UID: "1f86dd20-d255-4a9e-baa2-5dd9521bb79c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:53.678292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.678199    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir" pod "1f86dd20-d255-4a9e-baa2-5dd9521bb79c" (UID: "1f86dd20-d255-4a9e-baa2-5dd9521bb79c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:53.678420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.678400    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache" pod "1f86dd20-d255-4a9e-baa2-5dd9521bb79c" (UID: "1f86dd20-d255-4a9e-baa2-5dd9521bb79c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:53.678703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.678681    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:53.679120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.679098    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:53.679239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.679126    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:53.681603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.681565    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm" pod "1f86dd20-d255-4a9e-baa2-5dd9521bb79c" (UID: "1f86dd20-d255-4a9e-baa2-5dd9521bb79c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:53.681717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.681655    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd" pod "1f86dd20-d255-4a9e-baa2-5dd9521bb79c" (UID: "1f86dd20-d255-4a9e-baa2-5dd9521bb79c"). InnerVolumeSpecName "kube-api-access-xbmpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:53.681717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.681696    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs" pod "1f86dd20-d255-4a9e-baa2-5dd9521bb79c" (UID: "1f86dd20-d255-4a9e-baa2-5dd9521bb79c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:29:53.781223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.781177    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:53.781223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.781214    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:53.781223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.781229    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbmpd\" (UniqueName: \"kubernetes.io/projected/1f86dd20-d255-4a9e-baa2-5dd9521bb79c-kube-api-access-xbmpd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:53.929926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.929836    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" containerID="4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab" exitCode=0
Oct 30 12:29:53.929926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.929893    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"
Oct 30 12:29:53.930161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.929923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" event={"ID":"1f86dd20-d255-4a9e-baa2-5dd9521bb79c","Type":"ContainerDied","Data":"4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab"}
Oct 30 12:29:53.930161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.929970    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9" event={"ID":"1f86dd20-d255-4a9e-baa2-5dd9521bb79c","Type":"ContainerDied","Data":"67a9abe450075e519dea53dc2e0817831ede797229e53349c7d808ccba464c3a"}
Oct 30 12:29:53.930161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.929987    5060 scope.go:122] "RemoveContainer" containerID="4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab"
Oct 30 12:29:53.943998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.943959    5060 scope.go:122] "RemoveContainer" containerID="4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab"
Oct 30 12:29:53.944317 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:29:53.944298    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab\": container with ID starting with 4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab not found: ID does not exist" containerID="4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab"
Oct 30 12:29:53.944364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.944329    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab"} err="failed to get container status \"4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab\": rpc error: code = NotFound desc = could not find container \"4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab\": container with ID starting with 4fd6e9f62f8f06697b4a514a7fd3b636c846f461317109e22822e0b8455b48ab not found: ID does not exist"
Oct 30 12:29:53.990465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:53.990426    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"]
Oct 30 12:29:54.002327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.002295    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a1f322/test-model-a1f322-kserve-65558fb68f-wzts9"]
Oct 30 12:29:54.337578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.337542    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" path="/var/lib/kubelet/pods/1f86dd20-d255-4a9e-baa2-5dd9521bb79c/volumes"
Oct 30 12:29:54.465245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.465204    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:54.564474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.556245    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"]
Oct 30 12:29:54.740474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.720446    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:54.862246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.862211    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"]
Oct 30 12:29:54.863023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.863005    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" containerName="main"
Oct 30 12:29:54.863118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.863024    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" containerName="main"
Oct 30 12:29:54.863118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.863062    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" containerName="revoke-keys"
Oct 30 12:29:54.863118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.863068    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" containerName="revoke-keys"
Oct 30 12:29:54.863260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.863196    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f86dd20-d255-4a9e-baa2-5dd9521bb79c" containerName="main"
Oct 30 12:29:54.863260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.863210    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" containerName="revoke-keys"
Oct 30 12:29:54.868270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.868243    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:54.889532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.884340    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:29:54.890530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.890275    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"]
Oct 30 12:29:54.893274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.893238    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:29:54.916827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.916792    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:54.917145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.917123    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp62g\" (UniqueName: \"kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:54.917357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.917340    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:54.925790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.925768    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:29:54.926076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:54.926038    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:29:55.019134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.019047    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.019502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.019431    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp62g\" (UniqueName: \"kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.019753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.019734    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.019889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.019855    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.022901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.022874    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.043212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.043184    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp62g\" (UniqueName: \"kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.198562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.198527    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:55.289850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.289812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:55.521403 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:29:55.521353    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c98574_8d1f_4844_a8c1_62cdda93fbe0.slice/crio-5b97af0e05ff39d51c31764676b476b8d4eb00b6aed9229856b34d6c6aa7f31b WatchSource:0}: Error finding container 5b97af0e05ff39d51c31764676b476b8d4eb00b6aed9229856b34d6c6aa7f31b: Status 404 returned error can't find the container with id 5b97af0e05ff39d51c31764676b476b8d4eb00b6aed9229856b34d6c6aa7f31b
Oct 30 12:29:55.534119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.530718    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"]
Oct 30 12:29:55.958170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:55.958136    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh" event={"ID":"c2c98574-8d1f-4844-a8c1-62cdda93fbe0","Type":"ContainerStarted","Data":"5b97af0e05ff39d51c31764676b476b8d4eb00b6aed9229856b34d6c6aa7f31b"}
Oct 30 12:29:56.356720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:56.356676    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:56.968946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:56.968906    5060 generic.go:356] "Generic (PLEG): container finished" podID="c2c98574-8d1f-4844-a8c1-62cdda93fbe0" containerID="91aecbc407b6f8a2951f385f08083e3e6a4bc6e6eb2a8c8abfb8b1ba68b8c275" exitCode=0
Oct 30 12:29:56.969368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:56.968989    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh" event={"ID":"c2c98574-8d1f-4844-a8c1-62cdda93fbe0","Type":"ContainerDied","Data":"91aecbc407b6f8a2951f385f08083e3e6a4bc6e6eb2a8c8abfb8b1ba68b8c275"}
Oct 30 12:29:57.720853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:57.720822    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:29:57.737002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:57.736971    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:29:58.193634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.193610    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:58.226292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.226245    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:29:58.301917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.301713    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca\") pod \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") "
Oct 30 12:29:58.302412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.302387    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca\") pod \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") "
Oct 30 12:29:58.302738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.302717    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g\" (UniqueName: \"kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g\") pod \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\" (UID: \"c2c98574-8d1f-4844-a8c1-62cdda93fbe0\") "
Oct 30 12:29:58.312102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.312074    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g" pod "c2c98574-8d1f-4844-a8c1-62cdda93fbe0" (UID: "c2c98574-8d1f-4844-a8c1-62cdda93fbe0"). InnerVolumeSpecName "kube-api-access-gp62g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:58.312609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.312579    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca" pod "c2c98574-8d1f-4844-a8c1-62cdda93fbe0" (UID: "c2c98574-8d1f-4844-a8c1-62cdda93fbe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:29:58.318300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.318270    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca" pod "c2c98574-8d1f-4844-a8c1-62cdda93fbe0" (UID: "c2c98574-8d1f-4844-a8c1-62cdda93fbe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:29:58.409266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.409232    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp62g\" (UniqueName: \"kubernetes.io/projected/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-kube-api-access-gp62g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:58.409266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.409264    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:58.409545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:58.409276    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c98574-8d1f-4844-a8c1-62cdda93fbe0-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:29:59.010260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:59.010225    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"
Oct 30 12:29:59.010576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:59.010535    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh" event={"ID":"c2c98574-8d1f-4844-a8c1-62cdda93fbe0","Type":"ContainerDied","Data":"5b97af0e05ff39d51c31764676b476b8d4eb00b6aed9229856b34d6c6aa7f31b"}
Oct 30 12:29:59.010689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:59.010630    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b97af0e05ff39d51c31764676b476b8d4eb00b6aed9229856b34d6c6aa7f31b"
Oct 30 12:29:59.971756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:29:59.971720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:00.147828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.147786    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"]
Oct 30 12:30:00.149084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.149060    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c2c98574-8d1f-4844-a8c1-62cdda93fbe0" containerName="revoke-keys"
Oct 30 12:30:00.149155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.149088    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c98574-8d1f-4844-a8c1-62cdda93fbe0" containerName="revoke-keys"
Oct 30 12:30:00.149403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.149389    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c98574-8d1f-4844-a8c1-62cdda93fbe0" containerName="revoke-keys"
Oct 30 12:30:00.154136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.154110    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.155674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.155643    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"]
Oct 30 12:30:00.160527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.160505    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.163224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.163198    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"]
Oct 30 12:30:00.173265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.173231    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"]
Oct 30 12:30:00.176218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.176173    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.195966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.181909    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.216129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.208952    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:30:00.216129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.209063    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:30:00.216129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.209143    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:30:00.238486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.238029    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"]
Oct 30 12:30:00.246290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.246260    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.246894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.246863    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"]
Oct 30 12:30:00.273684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.257396    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"]
Oct 30 12:30:00.273884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.273757    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"]
Oct 30 12:30:00.301476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.284579    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.301476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.285792    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"]
Oct 30 12:30:00.301476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.293269    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.348562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348529    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fh6\" (UniqueName: \"kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6\") pod \"maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.348764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdslx\" (UniqueName: \"kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.348764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348704    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.348764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348744    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mqk\" (UniqueName: \"kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.348971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348773    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.348971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348815    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvfp\" (UniqueName: \"kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.348971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348833    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.348971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348885    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.348971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348913    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzp8\" (UniqueName: \"kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8\") pod \"maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.348971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.348932    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.361970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.361933    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"]
Oct 30 12:30:00.379809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.379776    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.380654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.380623    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"]
Oct 30 12:30:00.381536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.381251    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"]
Oct 30 12:30:00.381536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.381338    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"]
Oct 30 12:30:00.392878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.390278    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.392878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.390174    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"]
Oct 30 12:30:00.396329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.396299    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"]
Oct 30 12:30:00.402566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.402537    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.406866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.406822    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"]
Oct 30 12:30:00.424360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.424330    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.429057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.429027    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"]
Oct 30 12:30:00.435070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.435029    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"]
Oct 30 12:30:00.441731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.441706    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"]
Oct 30 12:30:00.446844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.446816    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.448042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.448020    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"]
Oct 30 12:30:00.456531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456442    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.456712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456544    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvzg\" (UniqueName: \"kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.456712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456585    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvn5\" (UniqueName: \"kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.456712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fh6\" (UniqueName: \"kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6\") pod \"maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.456915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.456915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456777    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdslx\" (UniqueName: \"kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.456915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456853    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.456915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456877    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg5sh\" (UniqueName: \"kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.457134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456936    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.457134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.456994    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.457134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457045    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7xw\" (UniqueName: \"kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.457134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457074    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.457134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457122    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mqk\" (UniqueName: \"kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457166    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457229    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457264    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457294    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvfp\" (UniqueName: \"kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457325    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457367    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.457489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457443    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.457971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457504    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkn9\" (UniqueName: \"kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.457971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457533    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89pc\" (UniqueName: \"kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.457971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457568    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzp8\" (UniqueName: \"kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8\") pod \"maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.457971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.457602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.458501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.458472    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.458675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.458529    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.459648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.459621    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.459821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.459655    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.462571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.460094    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.462571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.461685    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.469639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.469610    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"]
Oct 30 12:30:00.476289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.476266    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.477660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.477636    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"]
Oct 30 12:30:00.483565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.483534    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzp8\" (UniqueName: \"kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8\") pod \"maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.483727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.483711    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"]
Oct 30 12:30:00.485364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.485333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdslx\" (UniqueName: \"kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.487083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.487023    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fh6\" (UniqueName: \"kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6\") pod \"maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.491758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.491733    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.491892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.491782    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mqk\" (UniqueName: \"kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.492395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.492373    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvfp\" (UniqueName: \"kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.501207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.501176    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"]
Oct 30 12:30:00.510034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.510003    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"]
Oct 30 12:30:00.524199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.524165    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"]
Oct 30 12:30:00.530739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.530710    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"]
Oct 30 12:30:00.530878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.530839    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.533632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.533602    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"]
Oct 30 12:30:00.539145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.539124    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.542531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.542503    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"]
Oct 30 12:30:00.550312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.550284    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"]
Oct 30 12:30:00.551571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.551551    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:00.556966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.556945    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"]
Oct 30 12:30:00.557062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.557033    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:00.558966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.558937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7xw\" (UniqueName: \"kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.559101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.558977    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.559101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559032    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.559101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559074    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.559273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559112    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrd2\" (UniqueName: \"kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2\") pod \"maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.559273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559145    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.559273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559175    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.559273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559220    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.559273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.559257    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58rj\" (UniqueName: \"kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.560516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.560080    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.560909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.560884    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkn9\" (UniqueName: \"kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.560983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.560930    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89pc\" (UniqueName: \"kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.560983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.560966    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.561092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561028    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.561092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvzg\" (UniqueName: \"kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.561211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561093    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvn5\" (UniqueName: \"kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.561211 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561133    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.561327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561207    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhsfh\" (UniqueName: \"kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh\") pod \"maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.561327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.561327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.561563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561364    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftngq\" (UniqueName: \"kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.561563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561413    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.562501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.561440    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg5sh\" (UniqueName: \"kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.562620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.562508    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2wb\" (UniqueName: \"kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.562620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.562581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.563100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.563077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.563929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.563743    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.564374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.564351    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.565369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.565346    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.565524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.565499    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.567282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.566275    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.568740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.568716    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.573289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.573263    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"]
Oct 30 12:30:00.573518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.573496    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:00.583466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.583425    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.586757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.586726    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"]
Oct 30 12:30:00.590152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.590124    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg5sh\" (UniqueName: \"kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.590931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.590907    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkn9\" (UniqueName: \"kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.591964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.591936    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvzg\" (UniqueName: \"kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.594437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.594412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvn5\" (UniqueName: \"kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.594887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.594868    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.595488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.595446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7xw\" (UniqueName: \"kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.598306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.598258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.598571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.598551    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89pc\" (UniqueName: \"kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.607517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.607409    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"]
Oct 30 12:30:00.608769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.608739    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:00.642514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.638250    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"]
Oct 30 12:30:00.649115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.648860    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.673744    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"]
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.674031    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.674150    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.675684    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.675725    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqt8\" (UniqueName: \"kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.675779    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.675825    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.675866    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrd2\" (UniqueName: \"kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2\") pod \"maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.675939    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58rj\" (UniqueName: \"kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676040    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676121    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676190    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsfh\" (UniqueName: \"kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh\") pod \"maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676278    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676349    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftngq\" (UniqueName: \"kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676379    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qx7s\" (UniqueName: \"kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh52\" (UniqueName: \"kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.676480    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2wb\" (UniqueName: \"kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.677701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.677860    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:00.688577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.678644    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.695380    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.700045    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.709102    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.728890    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.730599    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.738533    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"]
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.741308    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsfh\" (UniqueName: \"kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh\") pod \"maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.741326    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2wb\" (UniqueName: \"kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.746397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.742798    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:00.756776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.756265    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:00.765889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.765863    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"]
Oct 30 12:30:00.766062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.765947    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.776575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.775385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrd2\" (UniqueName: \"kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2\") pod \"maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:00.776575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.775932    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58rj\" (UniqueName: \"kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:00.780503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780445    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.780661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780544    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.780797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74xc\" (UniqueName: \"kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc\") pod \"maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.780797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780750    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.780797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780780    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nmm\" (UniqueName: \"kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm\") pod \"maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.780964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780844    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qx7s\" (UniqueName: \"kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.780964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.780878    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh52\" (UniqueName: \"kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.781072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.781021    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.781072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.781051    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqt8\" (UniqueName: \"kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.781181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.781134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.782112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.782089    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.783226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.783199    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.784400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.784194    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:00.784400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.784385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.786998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.786976    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftngq\" (UniqueName: \"kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.797903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.797700    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:00.821069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.815301    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"]
Oct 30 12:30:00.834596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.831882    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"]
Oct 30 12:30:00.834596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.834523    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"]
Oct 30 12:30:00.838487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.836785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qx7s\" (UniqueName: \"kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.839311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.839236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqt8\" (UniqueName: \"kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8\") pod \"maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.842339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.841994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh52\" (UniqueName: \"kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52\") pod \"maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:00.845024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.844840    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:00.845138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.845064    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"]
Oct 30 12:30:00.845208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.845167    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:00.847442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.847288    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:00.850906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.850238    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"]
Oct 30 12:30:00.858194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.858167    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"]
Oct 30 12:30:00.858501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.858430    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:00.860686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.860634    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:00.867495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.867446    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"]
Oct 30 12:30:00.884748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.884714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.885041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.885017    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:00.885563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.885542    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.885711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.885696    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7r8\" (UniqueName: \"kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8\") pod \"maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:00.885905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.885886    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74xc\" (UniqueName: \"kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc\") pod \"maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.886066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.886039    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.886653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.886054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nmm\" (UniqueName: \"kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm\") pod \"maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.899802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.899768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.950296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.948561    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"]
Oct 30 12:30:00.957031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.956960    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nmm\" (UniqueName: \"kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm\") pod \"maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.961273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.960963    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74xc\" (UniqueName: \"kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc\") pod \"maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:00.964590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.964559    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:00.991876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.989296    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:00.991876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.989445    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7r8\" (UniqueName: \"kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8\") pod \"maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:00.992675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:00.992628    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:01.010965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.010735    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:01.017086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.017024    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7r8\" (UniqueName: \"kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8\") pod \"maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:01.047967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.047922    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:01.076204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.074259    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:01.192368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.192235    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:01.231486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.222533    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"]
Oct 30 12:30:01.281669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.281377    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:01.375251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.375200    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"]
Oct 30 12:30:01.438559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.436616    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"]
Oct 30 12:30:01.466809 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:01.466772    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed245e9_ef5a_4a2d_afda_2b1fc7bf9096.slice/crio-17dc4a38515d54a0bcfa7f8081683c12f82464b3dc2ce22d721ef43fe52d6758 WatchSource:0}: Error finding container 17dc4a38515d54a0bcfa7f8081683c12f82464b3dc2ce22d721ef43fe52d6758: Status 404 returned error can't find the container with id 17dc4a38515d54a0bcfa7f8081683c12f82464b3dc2ce22d721ef43fe52d6758
Oct 30 12:30:01.475058 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:01.475026    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a6acb4_68df_4d60_add2_101fe1ddf61e.slice/crio-928deb9e17c0677048bbb915ad906fb77897b1309930df0552ffdd3c666264d2 WatchSource:0}: Error finding container 928deb9e17c0677048bbb915ad906fb77897b1309930df0552ffdd3c666264d2: Status 404 returned error can't find the container with id 928deb9e17c0677048bbb915ad906fb77897b1309930df0552ffdd3c666264d2
Oct 30 12:30:01.871602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.871564    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:01.949317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.949269    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"]
Oct 30 12:30:01.988237 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:01.988193    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ba43b6_963c_4f0e_ab47_639eebd7497f.slice/crio-2d4b3c77b7743e40d0fd8104e9231e825479086808bf731ae4ae442135940d43 WatchSource:0}: Error finding container 2d4b3c77b7743e40d0fd8104e9231e825479086808bf731ae4ae442135940d43: Status 404 returned error can't find the container with id 2d4b3c77b7743e40d0fd8104e9231e825479086808bf731ae4ae442135940d43
Oct 30 12:30:01.989083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:01.989057    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"]
Oct 30 12:30:02.000730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.000702    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"]
Oct 30 12:30:02.016356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.016333    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"]
Oct 30 12:30:02.035338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.035309    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"]
Oct 30 12:30:02.039104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.039076    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"]
Oct 30 12:30:02.040712 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:02.040662    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a830ea6_1b84_4ddb_abe8_01830a9ce9e3.slice/crio-e2a5e6b0aa24e448a246cb66bcb0faec69c036d366cd9e763285113e34911a0a WatchSource:0}: Error finding container e2a5e6b0aa24e448a246cb66bcb0faec69c036d366cd9e763285113e34911a0a: Status 404 returned error can't find the container with id e2a5e6b0aa24e448a246cb66bcb0faec69c036d366cd9e763285113e34911a0a
Oct 30 12:30:02.041571 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:02.041439    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9c1455_fce3_4161_a671_e3e509d5d854.slice/crio-69b5ff5fa51a9fabcf0cc086b9959c6e237a421d51d7d23cbed077f2f52c9b3b WatchSource:0}: Error finding container 69b5ff5fa51a9fabcf0cc086b9959c6e237a421d51d7d23cbed077f2f52c9b3b: Status 404 returned error can't find the container with id 69b5ff5fa51a9fabcf0cc086b9959c6e237a421d51d7d23cbed077f2f52c9b3b
Oct 30 12:30:02.047557 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:02.047533    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144e4de0_8203_4e08_a384_f6a89b6627a5.slice/crio-80bfe18a35f29f341c1beb90bb388ee2ee805791ebd7ad0474f4383ad3887dbb WatchSource:0}: Error finding container 80bfe18a35f29f341c1beb90bb388ee2ee805791ebd7ad0474f4383ad3887dbb: Status 404 returned error can't find the container with id 80bfe18a35f29f341c1beb90bb388ee2ee805791ebd7ad0474f4383ad3887dbb
Oct 30 12:30:02.187954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.187916    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerStarted","Data":"e2a5e6b0aa24e448a246cb66bcb0faec69c036d366cd9e763285113e34911a0a"}
Oct 30 12:30:02.199664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.199626    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerStarted","Data":"8404ac62ae5e632cf3ae035f83afbde54bc30da80cf5cfd4976999687a5bdbfa"}
Oct 30 12:30:02.212780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.212756    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"]
Oct 30 12:30:02.212945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.212923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerStarted","Data":"1ca0380d15901d4921f7ed79327cf4662d847035053f8fcf126d3d662413334f"}
Oct 30 12:30:02.213066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.213052    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerStarted","Data":"80bfe18a35f29f341c1beb90bb388ee2ee805791ebd7ad0474f4383ad3887dbb"}
Oct 30 12:30:02.213166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.213152    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerStarted","Data":"2d4b3c77b7743e40d0fd8104e9231e825479086808bf731ae4ae442135940d43"}
Oct 30 12:30:02.213273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.213252    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" event={"ID":"d59ed183-da6d-4132-a925-3a23af0aeb20","Type":"ContainerStarted","Data":"d9650039aca80258c60309b5e485ddbf93b6d5c9110f3cf9c68501f021c314b8"}
Oct 30 12:30:02.236788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.236752    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerStarted","Data":"928deb9e17c0677048bbb915ad906fb77897b1309930df0552ffdd3c666264d2"}
Oct 30 12:30:02.251572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.251540    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"]
Oct 30 12:30:02.255468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.255418    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" event={"ID":"5c9c1455-fce3-4161-a671-e3e509d5d854","Type":"ContainerStarted","Data":"69b5ff5fa51a9fabcf0cc086b9959c6e237a421d51d7d23cbed077f2f52c9b3b"}
Oct 30 12:30:02.261190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.261114    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerStarted","Data":"17dc4a38515d54a0bcfa7f8081683c12f82464b3dc2ce22d721ef43fe52d6758"}
Oct 30 12:30:02.263809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.263705    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"]
Oct 30 12:30:02.281623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.281589    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"]
Oct 30 12:30:02.291202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.291174    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"]
Oct 30 12:30:02.316954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.314639    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" podStartSLOduration=2.314598652 podStartE2EDuration="2.314598652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.306934732 +0000 UTC m=+3606.101819229" watchObservedRunningTime="2026-10-30 12:30:02.314598652 +0000 UTC m=+3606.109483147"
Oct 30 12:30:02.325908 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:02.325862    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236a39f6_e955_4e84_b437_7592c13f8f76.slice/crio-a9c0bfcf11955ee8d37f395e4953e93e6b757e90f579d39f1538b0bd0fb9a972 WatchSource:0}: Error finding container a9c0bfcf11955ee8d37f395e4953e93e6b757e90f579d39f1538b0bd0fb9a972: Status 404 returned error can't find the container with id a9c0bfcf11955ee8d37f395e4953e93e6b757e90f579d39f1538b0bd0fb9a972
Oct 30 12:30:02.346326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.346297    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"]
Oct 30 12:30:02.346441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.346354    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"]
Oct 30 12:30:02.348084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.348059    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"]
Oct 30 12:30:02.358159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.355967    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"]
Oct 30 12:30:02.375860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.375836    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"]
Oct 30 12:30:02.405036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.405004    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"]
Oct 30 12:30:02.411638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.411610    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"]
Oct 30 12:30:02.418172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.418138    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"]
Oct 30 12:30:02.422387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:02.422359    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"]
Oct 30 12:30:02.500779 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:02.500751    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89f2ac9_4e23_41f2_a8ae_596ce7000751.slice/crio-4b3cb69ca096fd5156b4acba21b95e318638a7ba4e2542a17fd351764ea4444e WatchSource:0}: Error finding container 4b3cb69ca096fd5156b4acba21b95e318638a7ba4e2542a17fd351764ea4444e: Status 404 returned error can't find the container with id 4b3cb69ca096fd5156b4acba21b95e318638a7ba4e2542a17fd351764ea4444e
Oct 30 12:30:02.513443 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:02.513411    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4438f8b_b45c_4186_b63f_e9226d426ce8.slice/crio-e2e46e0718dd230a769e0829ec79831594c1d2cda1ac311f9aebf6bb24d23912 WatchSource:0}: Error finding container e2e46e0718dd230a769e0829ec79831594c1d2cda1ac311f9aebf6bb24d23912: Status 404 returned error can't find the container with id e2e46e0718dd230a769e0829ec79831594c1d2cda1ac311f9aebf6bb24d23912
Oct 30 12:30:03.298819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.298749    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerStarted","Data":"42094b881802314fecc3ddb41c6d00f701bae16c5837aa08f4f7ffa2fe9ea02e"}
Oct 30 12:30:03.301649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.301618    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerStarted","Data":"e2e46e0718dd230a769e0829ec79831594c1d2cda1ac311f9aebf6bb24d23912"}
Oct 30 12:30:03.304212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.304185    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerStarted","Data":"ddadef4fc0ead46059a1a7e28888cc57a331ee808c09b95985723ed0d0a68851"}
Oct 30 12:30:03.309224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.309197    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerStarted","Data":"dd36d781949d5f63e49ad691ed0ec6c4ed561c9903047ce29608398a0c31be5d"}
Oct 30 12:30:03.313758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.313732    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerStarted","Data":"d495b3c779e9df7e8b2fcb1d44eb3af57ee6862ce4ae8ad9042cce89e3a2bd6d"}
Oct 30 12:30:03.315749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.315721    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerStarted","Data":"a9c0bfcf11955ee8d37f395e4953e93e6b757e90f579d39f1538b0bd0fb9a972"}
Oct 30 12:30:03.317769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.317712    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerStarted","Data":"0fff5cdd387315945d2cf65b118ed48bfdabe74786c54322ffd391fc2134f82d"}
Oct 30 12:30:03.328140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.328114    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerStarted","Data":"b828d130a2b9556a229b376a0ba51a4571a23c1afb628c39f3e4da53f7d994d4"}
Oct 30 12:30:03.330616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.330593    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerStarted","Data":"b9a7f27eca6e5d2d80fa322db5959125dedc48f5c0cf945171a371c19f23cb13"}
Oct 30 12:30:03.330990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.330973    5060 scope.go:122] "RemoveContainer" containerID="b9a7f27eca6e5d2d80fa322db5959125dedc48f5c0cf945171a371c19f23cb13"
Oct 30 12:30:03.333310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.333292    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerStarted","Data":"42b611b95ed51fa5d3bbe2954ce4330d1dbd0d713aedb44fb220dfd30e3d6c4a"}
Oct 30 12:30:03.334813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.334793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" event={"ID":"a408b674-92df-4821-b568-7f6ee75c5d36","Type":"ContainerStarted","Data":"06d884301d6de4de91e661f63dec8ee0eccf257e7ecf67c271570d007f66cd36"}
Oct 30 12:30:03.335901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.335878    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerStarted","Data":"e35b930a00fcbb02f4803483688311767f27d90edcd94416857c983a7d2e0435"}
Oct 30 12:30:03.337398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.337369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerStarted","Data":"7b7087004cc48af5291c4c6641a65d914be531a4e8787fe57d2befc704b23407"}
Oct 30 12:30:03.339026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.338993    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerStarted","Data":"77ac4a577fe2a06efee70eed5d52589efbbc231ecbc5ee284f1fbf9cfad3dbd2"}
Oct 30 12:30:03.340644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.340619    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" event={"ID":"d59ed183-da6d-4132-a925-3a23af0aeb20","Type":"ContainerStarted","Data":"0fb452c670fcac8b7abb12d5a326b8ed2f73cbcfca46d8329bc5428ab75d7cb6"}
Oct 30 12:30:03.341946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.341928    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerStarted","Data":"03b37a8b367b38e290b6929f10a4787ac4e25047bba1b2ca584cb65998b2c612"}
Oct 30 12:30:03.343199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.343180    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerStarted","Data":"80a824ede2859c409d4048b89afb1c0e7499f1fbd5ce6805e4fd7c89372252e7"}
Oct 30 12:30:03.344426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.344407    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerStarted","Data":"02a639c1726b8e9c88db111321d0965d4415b894b7ec7499f6dca3d3db46213d"}
Oct 30 12:30:03.345917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.345894    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerStarted","Data":"956bff40f112ae51579d54504fd4c48be3684fe8207af2c2e0d0b1b3ad4b1046"}
Oct 30 12:30:03.347244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.347222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerStarted","Data":"505e49dd67a2351fe3039371c78026a0bfe1f9ade7f3857f31cba9f61274ee2b"}
Oct 30 12:30:03.348493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.348473    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerStarted","Data":"a593dcf6679734dade94d5c1568ed8bd264092037df794a6720f7b332f6ff37e"}
Oct 30 12:30:03.349925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.349906    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerStarted","Data":"4b3cb69ca096fd5156b4acba21b95e318638a7ba4e2542a17fd351764ea4444e"}
Oct 30 12:30:03.356398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.356360    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" podStartSLOduration=3.356350315 podStartE2EDuration="3.356350315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.351176219 +0000 UTC m=+3607.146060726" watchObservedRunningTime="2026-10-30 12:30:03.356350315 +0000 UTC m=+3607.151234810"
Oct 30 12:30:03.432638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.432578    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" podStartSLOduration=3.4325636409999998 podStartE2EDuration="3.432563641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.431425421 +0000 UTC m=+3607.226309940" watchObservedRunningTime="2026-10-30 12:30:03.432563641 +0000 UTC m=+3607.227448135"
Oct 30 12:30:03.476305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.476263    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" podStartSLOduration=3.47624916 podStartE2EDuration="3.47624916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.471416972 +0000 UTC m=+3607.266301539" watchObservedRunningTime="2026-10-30 12:30:03.47624916 +0000 UTC m=+3607.271133665"
Oct 30 12:30:03.514042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.513968    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" podStartSLOduration=3.513949941 podStartE2EDuration="3.513949941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.512181096 +0000 UTC m=+3607.307065587" watchObservedRunningTime="2026-10-30 12:30:03.513949941 +0000 UTC m=+3607.308834436"
Oct 30 12:30:03.552959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.552866    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" podStartSLOduration=3.552852882 podStartE2EDuration="3.552852882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.552431806 +0000 UTC m=+3607.347316301" watchObservedRunningTime="2026-10-30 12:30:03.552852882 +0000 UTC m=+3607.347737403"
Oct 30 12:30:03.594701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.594643    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" podStartSLOduration=3.594625191 podStartE2EDuration="3.594625191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.592804283 +0000 UTC m=+3607.387688778" watchObservedRunningTime="2026-10-30 12:30:03.594625191 +0000 UTC m=+3607.389509687"
Oct 30 12:30:03.745218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:03.745183    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:04.361942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.361840    5060 generic.go:356] "Generic (PLEG): container finished" podID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerID="b9a7f27eca6e5d2d80fa322db5959125dedc48f5c0cf945171a371c19f23cb13" exitCode=60
Oct 30 12:30:04.361942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.361921    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerDied","Data":"b9a7f27eca6e5d2d80fa322db5959125dedc48f5c0cf945171a371c19f23cb13"}
Oct 30 12:30:04.363874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.363843    5060 generic.go:356] "Generic (PLEG): container finished" podID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerID="42b611b95ed51fa5d3bbe2954ce4330d1dbd0d713aedb44fb220dfd30e3d6c4a" exitCode=60
Oct 30 12:30:04.364012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.363955    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerDied","Data":"42b611b95ed51fa5d3bbe2954ce4330d1dbd0d713aedb44fb220dfd30e3d6c4a"}
Oct 30 12:30:04.364381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.364360    5060 scope.go:122] "RemoveContainer" containerID="42b611b95ed51fa5d3bbe2954ce4330d1dbd0d713aedb44fb220dfd30e3d6c4a"
Oct 30 12:30:04.365837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.365794    5060 generic.go:356] "Generic (PLEG): container finished" podID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerID="77ac4a577fe2a06efee70eed5d52589efbbc231ecbc5ee284f1fbf9cfad3dbd2" exitCode=60
Oct 30 12:30:04.365837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.365821    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerDied","Data":"77ac4a577fe2a06efee70eed5d52589efbbc231ecbc5ee284f1fbf9cfad3dbd2"}
Oct 30 12:30:04.366191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.366170    5060 scope.go:122] "RemoveContainer" containerID="77ac4a577fe2a06efee70eed5d52589efbbc231ecbc5ee284f1fbf9cfad3dbd2"
Oct 30 12:30:04.368533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.368515    5060 generic.go:356] "Generic (PLEG): container finished" podID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerID="0fb452c670fcac8b7abb12d5a326b8ed2f73cbcfca46d8329bc5428ab75d7cb6" exitCode=60
Oct 30 12:30:04.368615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.368579    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" event={"ID":"d59ed183-da6d-4132-a925-3a23af0aeb20","Type":"ContainerDied","Data":"0fb452c670fcac8b7abb12d5a326b8ed2f73cbcfca46d8329bc5428ab75d7cb6"}
Oct 30 12:30:04.368922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.368902    5060 scope.go:122] "RemoveContainer" containerID="0fb452c670fcac8b7abb12d5a326b8ed2f73cbcfca46d8329bc5428ab75d7cb6"
Oct 30 12:30:04.370510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.370413    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerStarted","Data":"3ebbe5bdf4fc2223220de31c581ddbe328dc7f5d198f763bda931011a0d00586"}
Oct 30 12:30:04.372310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.372290    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerStarted","Data":"6231f5a4ab34caa28c1c0424d96fea190a1d7378401cb71d49dff69e26ee3485"}
Oct 30 12:30:04.374023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.374000    5060 generic.go:356] "Generic (PLEG): container finished" podID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerID="956bff40f112ae51579d54504fd4c48be3684fe8207af2c2e0d0b1b3ad4b1046" exitCode=60
Oct 30 12:30:04.374108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.374075    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerDied","Data":"956bff40f112ae51579d54504fd4c48be3684fe8207af2c2e0d0b1b3ad4b1046"}
Oct 30 12:30:04.374267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.374254    5060 scope.go:122] "RemoveContainer" containerID="956bff40f112ae51579d54504fd4c48be3684fe8207af2c2e0d0b1b3ad4b1046"
Oct 30 12:30:04.375773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.375752    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerStarted","Data":"bfff451c91374aa1041d7c35419aeb308e07258d668c42db385d6f3d33091903"}
Oct 30 12:30:04.380191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.379435    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerStarted","Data":"d012c70b57dcc99d50423c2ef21067c16cee068011cfd1e771ce1e75202af7d9"}
Oct 30 12:30:04.387980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.387492    5060 generic.go:356] "Generic (PLEG): container finished" podID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerID="0338f06e257f2ab87884d1f56f01c1346e32fe9d9c34129f53f860da9d03132c" exitCode=60
Oct 30 12:30:04.387980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.387615    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" event={"ID":"5c9c1455-fce3-4161-a671-e3e509d5d854","Type":"ContainerDied","Data":"0338f06e257f2ab87884d1f56f01c1346e32fe9d9c34129f53f860da9d03132c"}
Oct 30 12:30:04.387980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.387872    5060 scope.go:122] "RemoveContainer" containerID="0338f06e257f2ab87884d1f56f01c1346e32fe9d9c34129f53f860da9d03132c"
Oct 30 12:30:04.390260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.390238    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerStarted","Data":"f4ea74a37ce33c2b3e43f28e71eb3526e27eaecc732c8bdde88de05040d84b59"}
Oct 30 12:30:04.404423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.403775    5060 generic.go:356] "Generic (PLEG): container finished" podID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerID="ddadef4fc0ead46059a1a7e28888cc57a331ee808c09b95985723ed0d0a68851" exitCode=60
Oct 30 12:30:04.404423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.404043    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerDied","Data":"ddadef4fc0ead46059a1a7e28888cc57a331ee808c09b95985723ed0d0a68851"}
Oct 30 12:30:04.404423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.404205    5060 scope.go:122] "RemoveContainer" containerID="ddadef4fc0ead46059a1a7e28888cc57a331ee808c09b95985723ed0d0a68851"
Oct 30 12:30:04.440244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.440168    5060 generic.go:356] "Generic (PLEG): container finished" podID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerID="dd36d781949d5f63e49ad691ed0ec6c4ed561c9903047ce29608398a0c31be5d" exitCode=60
Oct 30 12:30:04.440331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.440277    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerDied","Data":"dd36d781949d5f63e49ad691ed0ec6c4ed561c9903047ce29608398a0c31be5d"}
Oct 30 12:30:04.440875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.440854    5060 scope.go:122] "RemoveContainer" containerID="dd36d781949d5f63e49ad691ed0ec6c4ed561c9903047ce29608398a0c31be5d"
Oct 30 12:30:04.449509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.449442    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" podStartSLOduration=4.449426743 podStartE2EDuration="4.449426743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.44864082 +0000 UTC m=+3608.243525318" watchObservedRunningTime="2026-10-30 12:30:04.449426743 +0000 UTC m=+3608.244311237"
Oct 30 12:30:04.449708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.449685    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerStarted","Data":"f1135a1127e214d6670936e69d9529b26a65c0e3a68e85de03706e44eaa87464"}
Oct 30 12:30:04.450138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.450113    5060 scope.go:122] "RemoveContainer" containerID="f1135a1127e214d6670936e69d9529b26a65c0e3a68e85de03706e44eaa87464"
Oct 30 12:30:04.453106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.453082    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerStarted","Data":"523bf185f4d57fd4e72e741f12729c44640b1eb9521b14f186d8198b6c98e6f2"}
Oct 30 12:30:04.459593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.459097    5060 generic.go:356] "Generic (PLEG): container finished" podID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerID="0fff5cdd387315945d2cf65b118ed48bfdabe74786c54322ffd391fc2134f82d" exitCode=60
Oct 30 12:30:04.459593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.459200    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerDied","Data":"0fff5cdd387315945d2cf65b118ed48bfdabe74786c54322ffd391fc2134f82d"}
Oct 30 12:30:04.459593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.459579    5060 scope.go:122] "RemoveContainer" containerID="0fff5cdd387315945d2cf65b118ed48bfdabe74786c54322ffd391fc2134f82d"
Oct 30 12:30:04.461974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.461948    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerStarted","Data":"8c3f47a94a6dff5eea4f2174a2e6835fdc16803ff1b79e6c4d246caf5e9497ac"}
Oct 30 12:30:04.574887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.574836    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" podStartSLOduration=4.5748231740000005 podStartE2EDuration="4.574823174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.574737225 +0000 UTC m=+3608.369621718" watchObservedRunningTime="2026-10-30 12:30:04.574823174 +0000 UTC m=+3608.369707667"
Oct 30 12:30:04.623930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.623814    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" podStartSLOduration=4.623789524 podStartE2EDuration="4.623789524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.615612145 +0000 UTC m=+3608.410496640" watchObservedRunningTime="2026-10-30 12:30:04.623789524 +0000 UTC m=+3608.418674018"
Oct 30 12:30:04.780883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.780825    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" podStartSLOduration=4.78081097 podStartE2EDuration="4.78081097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.77625757 +0000 UTC m=+3608.571142064" watchObservedRunningTime="2026-10-30 12:30:04.78081097 +0000 UTC m=+3608.575695465"
Oct 30 12:30:04.824810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.824740    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" podStartSLOduration=4.82472081 podStartE2EDuration="4.82472081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.815690482 +0000 UTC m=+3608.610574977" watchObservedRunningTime="2026-10-30 12:30:04.82472081 +0000 UTC m=+3608.619605308"
Oct 30 12:30:04.896029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.895937    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" podStartSLOduration=4.895922507 podStartE2EDuration="4.895922507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.894603552 +0000 UTC m=+3608.689488058" watchObservedRunningTime="2026-10-30 12:30:04.895922507 +0000 UTC m=+3608.690807055"
Oct 30 12:30:04.993660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:04.993599    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" podStartSLOduration=4.993579834 podStartE2EDuration="4.993579834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.98466241 +0000 UTC m=+3608.779546904" watchObservedRunningTime="2026-10-30 12:30:04.993579834 +0000 UTC m=+3608.788464328"
Oct 30 12:30:05.576384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.576216    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:05.624561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.614035    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerStarted","Data":"5eae8da47a1735efb53be2daf077ed6c00fdb784dbe3cb1a7fe43ec018519d05"}
Oct 30 12:30:05.726333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.725807    5060 generic.go:356] "Generic (PLEG): container finished" podID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerID="f4ea74a37ce33c2b3e43f28e71eb3526e27eaecc732c8bdde88de05040d84b59" exitCode=60
Oct 30 12:30:05.726333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.725927    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerDied","Data":"f4ea74a37ce33c2b3e43f28e71eb3526e27eaecc732c8bdde88de05040d84b59"}
Oct 30 12:30:05.727005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.726477    5060 scope.go:122] "RemoveContainer" containerID="f4ea74a37ce33c2b3e43f28e71eb3526e27eaecc732c8bdde88de05040d84b59"
Oct 30 12:30:05.731748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.731671    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerStarted","Data":"58f509593837b4798338d1266ff2a1085c855b8173c211b8988bd26dd621374a"}
Oct 30 12:30:05.747896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.742249    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" podStartSLOduration=5.74222738 podStartE2EDuration="5.74222738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:05.667938442 +0000 UTC m=+3609.462822961" watchObservedRunningTime="2026-10-30 12:30:05.74222738 +0000 UTC m=+3609.537111877"
Oct 30 12:30:05.794539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.793748    5060 generic.go:356] "Generic (PLEG): container finished" podID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerID="f1135a1127e214d6670936e69d9529b26a65c0e3a68e85de03706e44eaa87464" exitCode=60
Oct 30 12:30:05.794539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.793915    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerDied","Data":"f1135a1127e214d6670936e69d9529b26a65c0e3a68e85de03706e44eaa87464"}
Oct 30 12:30:05.799637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.798531    5060 generic.go:356] "Generic (PLEG): container finished" podID="236a39f6-e955-4e84-b437-7592c13f8f76" containerID="523bf185f4d57fd4e72e741f12729c44640b1eb9521b14f186d8198b6c98e6f2" exitCode=60
Oct 30 12:30:05.799637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.798715    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerDied","Data":"523bf185f4d57fd4e72e741f12729c44640b1eb9521b14f186d8198b6c98e6f2"}
Oct 30 12:30:05.799637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.799044    5060 scope.go:122] "RemoveContainer" containerID="523bf185f4d57fd4e72e741f12729c44640b1eb9521b14f186d8198b6c98e6f2"
Oct 30 12:30:05.807996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.807392    5060 generic.go:356] "Generic (PLEG): container finished" podID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerID="8c3f47a94a6dff5eea4f2174a2e6835fdc16803ff1b79e6c4d246caf5e9497ac" exitCode=60
Oct 30 12:30:05.807996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.807519    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerDied","Data":"8c3f47a94a6dff5eea4f2174a2e6835fdc16803ff1b79e6c4d246caf5e9497ac"}
Oct 30 12:30:05.807996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.807885    5060 scope.go:122] "RemoveContainer" containerID="8c3f47a94a6dff5eea4f2174a2e6835fdc16803ff1b79e6c4d246caf5e9497ac"
Oct 30 12:30:05.814492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.813744    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerStarted","Data":"5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767"}
Oct 30 12:30:05.819841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.817014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" event={"ID":"a408b674-92df-4821-b568-7f6ee75c5d36","Type":"ContainerStarted","Data":"679d5f8e0f5932bbd5dd7089f14ab0ee286aa37bd7ce635d56ca29c2f633b240"}
Oct 30 12:30:05.832928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.830837    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerStarted","Data":"ffa0229c926f1a47398c8f5819376a22fb927391290b2eb2b3dfc5dc486cb0b2"}
Oct 30 12:30:05.832928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.832610    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" podStartSLOduration=5.8325870779999995 podStartE2EDuration="5.832587078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:05.818218919 +0000 UTC m=+3609.613103438" watchObservedRunningTime="2026-10-30 12:30:05.832587078 +0000 UTC m=+3609.627471572"
Oct 30 12:30:05.834629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.834602    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerStarted","Data":"48935a5d4eb8250ac06a8bcc1fb0f85897d1d50fb813b2f7baa517deea922259"}
Oct 30 12:30:05.836772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.836750    5060 generic.go:356] "Generic (PLEG): container finished" podID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerID="3ebbe5bdf4fc2223220de31c581ddbe328dc7f5d198f763bda931011a0d00586" exitCode=60
Oct 30 12:30:05.836869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.836808    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerDied","Data":"3ebbe5bdf4fc2223220de31c581ddbe328dc7f5d198f763bda931011a0d00586"}
Oct 30 12:30:05.837184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.837166    5060 scope.go:122] "RemoveContainer" containerID="3ebbe5bdf4fc2223220de31c581ddbe328dc7f5d198f763bda931011a0d00586"
Oct 30 12:30:05.839312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.839286    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerStarted","Data":"85558e47f0d18fa419590de77a43f97c66e45ff436fc623dcc6e292eddf01c0b"}
Oct 30 12:30:05.839600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.839581    5060 scope.go:122] "RemoveContainer" containerID="85558e47f0d18fa419590de77a43f97c66e45ff436fc623dcc6e292eddf01c0b"
Oct 30 12:30:05.845237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.844645    5060 generic.go:356] "Generic (PLEG): container finished" podID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerID="6231f5a4ab34caa28c1c0424d96fea190a1d7378401cb71d49dff69e26ee3485" exitCode=60
Oct 30 12:30:05.845237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.844753    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerDied","Data":"6231f5a4ab34caa28c1c0424d96fea190a1d7378401cb71d49dff69e26ee3485"}
Oct 30 12:30:05.846762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.846667    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6c1d179-d831-4a74-a377-1d549e622017" containerID="bfff451c91374aa1041d7c35419aeb308e07258d668c42db385d6f3d33091903" exitCode=60
Oct 30 12:30:05.846762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.846695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerDied","Data":"bfff451c91374aa1041d7c35419aeb308e07258d668c42db385d6f3d33091903"}
Oct 30 12:30:05.847512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.847052    5060 scope.go:122] "RemoveContainer" containerID="6231f5a4ab34caa28c1c0424d96fea190a1d7378401cb71d49dff69e26ee3485"
Oct 30 12:30:05.849440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.849415    5060 generic.go:356] "Generic (PLEG): container finished" podID="292ec118-23ed-4030-843d-8a906ff083c7" containerID="d012c70b57dcc99d50423c2ef21067c16cee068011cfd1e771ce1e75202af7d9" exitCode=60
Oct 30 12:30:05.849552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.849517    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerDied","Data":"d012c70b57dcc99d50423c2ef21067c16cee068011cfd1e771ce1e75202af7d9"}
Oct 30 12:30:05.849847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.849830    5060 scope.go:122] "RemoveContainer" containerID="d012c70b57dcc99d50423c2ef21067c16cee068011cfd1e771ce1e75202af7d9"
Oct 30 12:30:05.850721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.850704    5060 scope.go:122] "RemoveContainer" containerID="bfff451c91374aa1041d7c35419aeb308e07258d668c42db385d6f3d33091903"
Oct 30 12:30:05.910698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.910633    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" podStartSLOduration=5.910613759 podStartE2EDuration="5.910613759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:05.900794646 +0000 UTC m=+3609.695679141" watchObservedRunningTime="2026-10-30 12:30:05.910613759 +0000 UTC m=+3609.705498254"
Oct 30 12:30:05.987178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:05.987073    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" podStartSLOduration=5.98706087 podStartE2EDuration="5.98706087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:05.98385628 +0000 UTC m=+3609.778740775" watchObservedRunningTime="2026-10-30 12:30:05.98706087 +0000 UTC m=+3609.781945363"
Oct 30 12:30:06.128984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.128912    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" podStartSLOduration=6.128889007 podStartE2EDuration="6.128889007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:06.110834713 +0000 UTC m=+3609.905719214" watchObservedRunningTime="2026-10-30 12:30:06.128889007 +0000 UTC m=+3609.923773503"
Oct 30 12:30:06.225511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.222839    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" podStartSLOduration=6.222819483 podStartE2EDuration="6.222819483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:06.201039098 +0000 UTC m=+3609.995923594" watchObservedRunningTime="2026-10-30 12:30:06.222819483 +0000 UTC m=+3610.017703978"
Oct 30 12:30:06.541854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.539962    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:06.716548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.710798    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:06.861622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.861584    5060 generic.go:356] "Generic (PLEG): container finished" podID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerID="85558e47f0d18fa419590de77a43f97c66e45ff436fc623dcc6e292eddf01c0b" exitCode=60
Oct 30 12:30:06.861803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.861691    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerDied","Data":"85558e47f0d18fa419590de77a43f97c66e45ff436fc623dcc6e292eddf01c0b"}
Oct 30 12:30:06.864285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.864259    5060 generic.go:356] "Generic (PLEG): container finished" podID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerID="2cda6514c7f099ef1d81bd08bcd78191815ff8308beedb4d3c2a7aa01d8d8f82" exitCode=60
Oct 30 12:30:06.864437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.864363    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerDied","Data":"2cda6514c7f099ef1d81bd08bcd78191815ff8308beedb4d3c2a7aa01d8d8f82"}
Oct 30 12:30:06.864437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.864415    5060 scope.go:122] "RemoveContainer" containerID="956bff40f112ae51579d54504fd4c48be3684fe8207af2c2e0d0b1b3ad4b1046"
Oct 30 12:30:06.864952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.864934    5060 scope.go:122] "RemoveContainer" containerID="2cda6514c7f099ef1d81bd08bcd78191815ff8308beedb4d3c2a7aa01d8d8f82"
Oct 30 12:30:06.865281 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.865259    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22_odh-ai-gateway-infra(f0a6acb4-68df-4d60-add2-101fe1ddf61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e"
Oct 30 12:30:06.867765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.867743    5060 generic.go:356] "Generic (PLEG): container finished" podID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerID="031446c081f23c7f2f4f97e5d6930b5a840e9b18f9385accfa45c980136eef23" exitCode=60
Oct 30 12:30:06.867886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.867781    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" event={"ID":"5c9c1455-fce3-4161-a671-e3e509d5d854","Type":"ContainerDied","Data":"031446c081f23c7f2f4f97e5d6930b5a840e9b18f9385accfa45c980136eef23"}
Oct 30 12:30:06.868227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.868207    5060 scope.go:122] "RemoveContainer" containerID="031446c081f23c7f2f4f97e5d6930b5a840e9b18f9385accfa45c980136eef23"
Oct 30 12:30:06.868693 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.868667    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz_odh-ai-gateway-infra(5c9c1455-fce3-4161-a671-e3e509d5d854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854"
Oct 30 12:30:06.869660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.869635    5060 generic.go:356] "Generic (PLEG): container finished" podID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerID="5eae8da47a1735efb53be2daf077ed6c00fdb784dbe3cb1a7fe43ec018519d05" exitCode=60
Oct 30 12:30:06.869750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.869730    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerDied","Data":"5eae8da47a1735efb53be2daf077ed6c00fdb784dbe3cb1a7fe43ec018519d05"}
Oct 30 12:30:06.869948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.869934    5060 scope.go:122] "RemoveContainer" containerID="5eae8da47a1735efb53be2daf077ed6c00fdb784dbe3cb1a7fe43ec018519d05"
Oct 30 12:30:06.871586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.871569    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerID="58f509593837b4798338d1266ff2a1085c855b8173c211b8988bd26dd621374a" exitCode=60
Oct 30 12:30:06.871673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.871651    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerDied","Data":"58f509593837b4798338d1266ff2a1085c855b8173c211b8988bd26dd621374a"}
Oct 30 12:30:06.871900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.871883    5060 scope.go:122] "RemoveContainer" containerID="58f509593837b4798338d1266ff2a1085c855b8173c211b8988bd26dd621374a"
Oct 30 12:30:06.908816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.904764    5060 generic.go:356] "Generic (PLEG): container finished" podID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerID="08e99d7dfc7bfbb224abc83f44bf2e046abbbfdad517448aeaf471ea29d8c8e2" exitCode=60
Oct 30 12:30:06.908816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.904918    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerDied","Data":"08e99d7dfc7bfbb224abc83f44bf2e046abbbfdad517448aeaf471ea29d8c8e2"}
Oct 30 12:30:06.908816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.905539    5060 scope.go:122] "RemoveContainer" containerID="08e99d7dfc7bfbb224abc83f44bf2e046abbbfdad517448aeaf471ea29d8c8e2"
Oct 30 12:30:06.908816 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.906040    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4_odh-ai-gateway-infra(3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096"
Oct 30 12:30:06.924822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.924785    5060 generic.go:356] "Generic (PLEG): container finished" podID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerID="a22a97668c61911043e46e59d411dafb0340a5b184bd3afeb9802a7937cdc8d2" exitCode=60
Oct 30 12:30:06.925044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.924868    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerDied","Data":"a22a97668c61911043e46e59d411dafb0340a5b184bd3afeb9802a7937cdc8d2"}
Oct 30 12:30:06.925292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.925274    5060 scope.go:122] "RemoveContainer" containerID="a22a97668c61911043e46e59d411dafb0340a5b184bd3afeb9802a7937cdc8d2"
Oct 30 12:30:06.925648 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.925628    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp_odh-ai-gateway-infra(4a830ea6-1b84-4ddb-abe8-01830a9ce9e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3"
Oct 30 12:30:06.928342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.928311    5060 generic.go:356] "Generic (PLEG): container finished" podID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerID="4f3098c99e36c769a525ec21f3543e2d3d57954d2d816cc7849a3f9eb8648257" exitCode=60
Oct 30 12:30:06.928518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.928357    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerDied","Data":"4f3098c99e36c769a525ec21f3543e2d3d57954d2d816cc7849a3f9eb8648257"}
Oct 30 12:30:06.928904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.928886    5060 scope.go:122] "RemoveContainer" containerID="4f3098c99e36c769a525ec21f3543e2d3d57954d2d816cc7849a3f9eb8648257"
Oct 30 12:30:06.929370 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.929283    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7_odh-ai-gateway-infra(0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.931260    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerDied","Data":"39ae35a0a354f18d6891de377e04169be662d6f011e1dfb118c0f0268caf2fa2"}
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.931175    5060 generic.go:356] "Generic (PLEG): container finished" podID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerID="39ae35a0a354f18d6891de377e04169be662d6f011e1dfb118c0f0268caf2fa2" exitCode=60
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.932078    5060 scope.go:122] "RemoveContainer" containerID="39ae35a0a354f18d6891de377e04169be662d6f011e1dfb118c0f0268caf2fa2"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.932317    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr_odh-ai-gateway-infra(e03a2654-376e-4fb8-a9fd-2b5bf73e3f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.934872    5060 generic.go:356] "Generic (PLEG): container finished" podID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerID="5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767" exitCode=60
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.934948    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerDied","Data":"5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767"}
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.935299    5060 scope.go:122] "RemoveContainer" containerID="5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.935662    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc_odh-ai-gateway-infra(4d41c30a-79d8-463b-94a6-4da554dbf161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.938267    5060 generic.go:356] "Generic (PLEG): container finished" podID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerID="6bd63bea1ae3d741bc674f23dc56370570710fcbb1f6e1893cc034882158fe9c" exitCode=60
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.938344    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerDied","Data":"6bd63bea1ae3d741bc674f23dc56370570710fcbb1f6e1893cc034882158fe9c"}
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.938656    5060 scope.go:122] "RemoveContainer" containerID="6bd63bea1ae3d741bc674f23dc56370570710fcbb1f6e1893cc034882158fe9c"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.938920    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb_odh-ai-gateway-infra(144e4de0-8203-4e08-a384-f6a89b6627a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5"
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.940078    5060 generic.go:356] "Generic (PLEG): container finished" podID="a408b674-92df-4821-b568-7f6ee75c5d36" containerID="679d5f8e0f5932bbd5dd7089f14ab0ee286aa37bd7ce635d56ca29c2f633b240" exitCode=60
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.940149    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" event={"ID":"a408b674-92df-4821-b568-7f6ee75c5d36","Type":"ContainerDied","Data":"679d5f8e0f5932bbd5dd7089f14ab0ee286aa37bd7ce635d56ca29c2f633b240"}
Oct 30 12:30:06.940552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.940425    5060 scope.go:122] "RemoveContainer" containerID="679d5f8e0f5932bbd5dd7089f14ab0ee286aa37bd7ce635d56ca29c2f633b240"
Oct 30 12:30:06.941432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.941246    5060 scope.go:122] "RemoveContainer" containerID="0338f06e257f2ab87884d1f56f01c1346e32fe9d9c34129f53f860da9d03132c"
Oct 30 12:30:06.942235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.942133    5060 generic.go:356] "Generic (PLEG): container finished" podID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerID="ffa0229c926f1a47398c8f5819376a22fb927391290b2eb2b3dfc5dc486cb0b2" exitCode=60
Oct 30 12:30:06.942235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.942219    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerDied","Data":"ffa0229c926f1a47398c8f5819376a22fb927391290b2eb2b3dfc5dc486cb0b2"}
Oct 30 12:30:06.942528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.942507    5060 scope.go:122] "RemoveContainer" containerID="ffa0229c926f1a47398c8f5819376a22fb927391290b2eb2b3dfc5dc486cb0b2"
Oct 30 12:30:06.944009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.943986    5060 generic.go:356] "Generic (PLEG): container finished" podID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerID="48935a5d4eb8250ac06a8bcc1fb0f85897d1d50fb813b2f7baa517deea922259" exitCode=60
Oct 30 12:30:06.944104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.944044    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerDied","Data":"48935a5d4eb8250ac06a8bcc1fb0f85897d1d50fb813b2f7baa517deea922259"}
Oct 30 12:30:06.944285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.944269    5060 scope.go:122] "RemoveContainer" containerID="48935a5d4eb8250ac06a8bcc1fb0f85897d1d50fb813b2f7baa517deea922259"
Oct 30 12:30:06.946595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.946575    5060 generic.go:356] "Generic (PLEG): container finished" podID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerID="11c743e8464c3b001d30257d0c8580159e48d1eb40d66ac46a66f2f9aa5e8de2" exitCode=60
Oct 30 12:30:06.946692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.946630    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerDied","Data":"11c743e8464c3b001d30257d0c8580159e48d1eb40d66ac46a66f2f9aa5e8de2"}
Oct 30 12:30:06.946889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.946872    5060 scope.go:122] "RemoveContainer" containerID="11c743e8464c3b001d30257d0c8580159e48d1eb40d66ac46a66f2f9aa5e8de2"
Oct 30 12:30:06.947198 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.947175    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2_odh-ai-gateway-infra(e3ba43b6-963c-4f0e-ab47-639eebd7497f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f"
Oct 30 12:30:06.952625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.952246    5060 scope.go:122] "RemoveContainer" containerID="9f1a3dd280bb062d6a313b7005c9476ddd3ea65c28c1f16098aa75f23e22d2fb"
Oct 30 12:30:06.952625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.952556    5060 generic.go:356] "Generic (PLEG): container finished" podID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerID="9f1a3dd280bb062d6a313b7005c9476ddd3ea65c28c1f16098aa75f23e22d2fb" exitCode=60
Oct 30 12:30:06.952625 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:06.952575    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5_odh-ai-gateway-infra(d59ed183-da6d-4132-a925-3a23af0aeb20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20"
Oct 30 12:30:06.952625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:06.952610    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" event={"ID":"d59ed183-da6d-4132-a925-3a23af0aeb20","Type":"ContainerDied","Data":"9f1a3dd280bb062d6a313b7005c9476ddd3ea65c28c1f16098aa75f23e22d2fb"}
Oct 30 12:30:07.460681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.460562    5060 scope.go:122] "RemoveContainer" containerID="ddadef4fc0ead46059a1a7e28888cc57a331ee808c09b95985723ed0d0a68851"
Oct 30 12:30:07.540770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.540670    5060 scope.go:122] "RemoveContainer" containerID="dd36d781949d5f63e49ad691ed0ec6c4ed561c9903047ce29608398a0c31be5d"
Oct 30 12:30:07.569630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.569601    5060 scope.go:122] "RemoveContainer" containerID="f1135a1127e214d6670936e69d9529b26a65c0e3a68e85de03706e44eaa87464"
Oct 30 12:30:07.638222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.638198    5060 scope.go:122] "RemoveContainer" containerID="0fff5cdd387315945d2cf65b118ed48bfdabe74786c54322ffd391fc2134f82d"
Oct 30 12:30:07.655210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.655144    5060 scope.go:122] "RemoveContainer" containerID="b9a7f27eca6e5d2d80fa322db5959125dedc48f5c0cf945171a371c19f23cb13"
Oct 30 12:30:07.674742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.674718    5060 scope.go:122] "RemoveContainer" containerID="42b611b95ed51fa5d3bbe2954ce4330d1dbd0d713aedb44fb220dfd30e3d6c4a"
Oct 30 12:30:07.745526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.745501    5060 scope.go:122] "RemoveContainer" containerID="77ac4a577fe2a06efee70eed5d52589efbbc231ecbc5ee284f1fbf9cfad3dbd2"
Oct 30 12:30:07.764152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.764126    5060 scope.go:122] "RemoveContainer" containerID="0fb452c670fcac8b7abb12d5a326b8ed2f73cbcfca46d8329bc5428ab75d7cb6"
Oct 30 12:30:07.874297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:07.873886    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:08.027411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.027233    5060 scope.go:122] "RemoveContainer" containerID="5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767"
Oct 30 12:30:08.027833 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.027798    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc_odh-ai-gateway-infra(4d41c30a-79d8-463b-94a6-4da554dbf161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161"
Oct 30 12:30:08.033333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.033297    5060 scope.go:122] "RemoveContainer" containerID="6bd63bea1ae3d741bc674f23dc56370570710fcbb1f6e1893cc034882158fe9c"
Oct 30 12:30:08.033772 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.033733    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb_odh-ai-gateway-infra(144e4de0-8203-4e08-a384-f6a89b6627a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5"
Oct 30 12:30:08.036272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.036248    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerStarted","Data":"c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364"}
Oct 30 12:30:08.039046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.039018    5060 scope.go:122] "RemoveContainer" containerID="11c743e8464c3b001d30257d0c8580159e48d1eb40d66ac46a66f2f9aa5e8de2"
Oct 30 12:30:08.039382 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.039358    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2_odh-ai-gateway-infra(e3ba43b6-963c-4f0e-ab47-639eebd7497f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f"
Oct 30 12:30:08.039866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.039850    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"]
Oct 30 12:30:08.040776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.040758    5060 scope.go:122] "RemoveContainer" containerID="9f1a3dd280bb062d6a313b7005c9476ddd3ea65c28c1f16098aa75f23e22d2fb"
Oct 30 12:30:08.041049 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.041026    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5_odh-ai-gateway-infra(d59ed183-da6d-4132-a925-3a23af0aeb20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20"
Oct 30 12:30:08.042815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.042791    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerStarted","Data":"b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7"}
Oct 30 12:30:08.046128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.046104    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerStarted","Data":"9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25"}
Oct 30 12:30:08.048147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.048130    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerStarted","Data":"0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e"}
Oct 30 12:30:08.050939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.050914    5060 scope.go:122] "RemoveContainer" containerID="2cda6514c7f099ef1d81bd08bcd78191815ff8308beedb4d3c2a7aa01d8d8f82"
Oct 30 12:30:08.051210 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.051188    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22_odh-ai-gateway-infra(f0a6acb4-68df-4d60-add2-101fe1ddf61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e"
Oct 30 12:30:08.051577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.051559    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8994e8b-j82k6"]
Oct 30 12:30:08.053750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.053729    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerStarted","Data":"f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96"}
Oct 30 12:30:08.055748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.055726    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerStarted","Data":"b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"}
Oct 30 12:30:08.055999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.055984    5060 scope.go:122] "RemoveContainer" containerID="b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"
Oct 30 12:30:08.056360 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.056321    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7_odh-ai-gateway-infra(292ec118-23ed-4030-843d-8a906ff083c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" podUID="292ec118-23ed-4030-843d-8a906ff083c7"
Oct 30 12:30:08.057752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.057736    5060 scope.go:122] "RemoveContainer" containerID="031446c081f23c7f2f4f97e5d6930b5a840e9b18f9385accfa45c980136eef23"
Oct 30 12:30:08.057952 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.057931    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz_odh-ai-gateway-infra(5c9c1455-fce3-4161-a671-e3e509d5d854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854"
Oct 30 12:30:08.059382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.059365    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerStarted","Data":"13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe"}
Oct 30 12:30:08.062494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.062472    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerStarted","Data":"0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b"}
Oct 30 12:30:08.064645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.064623    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerStarted","Data":"e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4"}
Oct 30 12:30:08.066844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.066826    5060 scope.go:122] "RemoveContainer" containerID="08e99d7dfc7bfbb224abc83f44bf2e046abbbfdad517448aeaf471ea29d8c8e2"
Oct 30 12:30:08.067046 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.067025    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4_odh-ai-gateway-infra(3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096"
Oct 30 12:30:08.068732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.068701    5060 scope.go:122] "RemoveContainer" containerID="a22a97668c61911043e46e59d411dafb0340a5b184bd3afeb9802a7937cdc8d2"
Oct 30 12:30:08.069008 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.068986    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp_odh-ai-gateway-infra(4a830ea6-1b84-4ddb-abe8-01830a9ce9e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3"
Oct 30 12:30:08.071082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.071034    5060 scope.go:122] "RemoveContainer" containerID="4f3098c99e36c769a525ec21f3543e2d3d57954d2d816cc7849a3f9eb8648257"
Oct 30 12:30:08.071252 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.071235    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7_odh-ai-gateway-infra(0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8"
Oct 30 12:30:08.073144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.073123    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerStarted","Data":"89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066"}
Oct 30 12:30:08.075904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.075887    5060 scope.go:122] "RemoveContainer" containerID="39ae35a0a354f18d6891de377e04169be662d6f011e1dfb118c0f0268caf2fa2"
Oct 30 12:30:08.076133 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:08.076114    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr_odh-ai-gateway-infra(e03a2654-376e-4fb8-a9fd-2b5bf73e3f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11"
Oct 30 12:30:08.125721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.125683    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerStarted","Data":"36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488"}
Oct 30 12:30:08.340617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.340529    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3582bc-1d47-4be4-8b9a-18d7347f667d" path="/var/lib/kubelet/pods/bc3582bc-1d47-4be4-8b9a-18d7347f667d/volumes"
Oct 30 12:30:08.611539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.610199    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:08.912477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:08.908324    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" podStartSLOduration=8.908299952 podStartE2EDuration="8.908299952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:08.898779407 +0000 UTC m=+3612.693663907" watchObservedRunningTime="2026-10-30 12:30:08.908299952 +0000 UTC m=+3612.703184447"
Oct 30 12:30:09.140529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.140410    5060 generic.go:356] "Generic (PLEG): container finished" podID="236a39f6-e955-4e84-b437-7592c13f8f76" containerID="89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066" exitCode=60
Oct 30 12:30:09.140705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.140563    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerDied","Data":"89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066"}
Oct 30 12:30:09.140705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.140630    5060 scope.go:122] "RemoveContainer" containerID="523bf185f4d57fd4e72e741f12729c44640b1eb9521b14f186d8198b6c98e6f2"
Oct 30 12:30:09.140985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.140964    5060 scope.go:122] "RemoveContainer" containerID="89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066"
Oct 30 12:30:09.141852 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.141483    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx_odh-ai-gateway-infra(236a39f6-e955-4e84-b437-7592c13f8f76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" podUID="236a39f6-e955-4e84-b437-7592c13f8f76"
Oct 30 12:30:09.144123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.143423    5060 generic.go:356] "Generic (PLEG): container finished" podID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerID="36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488" exitCode=60
Oct 30 12:30:09.144123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.143510    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerDied","Data":"36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488"}
Oct 30 12:30:09.144123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.143756    5060 scope.go:122] "RemoveContainer" containerID="36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488"
Oct 30 12:30:09.144123 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.144075    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr_odh-ai-gateway-infra(0a89196e-f4d8-436e-8bce-144bd3164a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03"
Oct 30 12:30:09.145543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.145517    5060 generic.go:356] "Generic (PLEG): container finished" podID="a408b674-92df-4821-b568-7f6ee75c5d36" containerID="b6a6b8bf2427ec570a7cabf51c436b65ecc0d6d90a7fbdc2610e8651c5a2051a" exitCode=60
Oct 30 12:30:09.145666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.145603    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" event={"ID":"a408b674-92df-4821-b568-7f6ee75c5d36","Type":"ContainerDied","Data":"b6a6b8bf2427ec570a7cabf51c436b65ecc0d6d90a7fbdc2610e8651c5a2051a"}
Oct 30 12:30:09.145911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.145893    5060 scope.go:122] "RemoveContainer" containerID="b6a6b8bf2427ec570a7cabf51c436b65ecc0d6d90a7fbdc2610e8651c5a2051a"
Oct 30 12:30:09.146265 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.146243    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z_odh-ai-gateway-infra(a408b674-92df-4821-b568-7f6ee75c5d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" podUID="a408b674-92df-4821-b568-7f6ee75c5d36"
Oct 30 12:30:09.147811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.147787    5060 generic.go:356] "Generic (PLEG): container finished" podID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerID="c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364" exitCode=60
Oct 30 12:30:09.147891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.147860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerDied","Data":"c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364"}
Oct 30 12:30:09.148050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.148035    5060 scope.go:122] "RemoveContainer" containerID="c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364"
Oct 30 12:30:09.148320 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.148300    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8_odh-ai-gateway-infra(4e468f83-2204-4f1d-9a08-f41115e4ef86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86"
Oct 30 12:30:09.150755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.150720    5060 generic.go:356] "Generic (PLEG): container finished" podID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerID="222068f2daa46fd328d01ec2f3cd89e0121a4bf7ccd963cd01c0553d7f55fd4f" exitCode=60
Oct 30 12:30:09.151771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.151242    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerDied","Data":"222068f2daa46fd328d01ec2f3cd89e0121a4bf7ccd963cd01c0553d7f55fd4f"}
Oct 30 12:30:09.151771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.151392    5060 scope.go:122] "RemoveContainer" containerID="222068f2daa46fd328d01ec2f3cd89e0121a4bf7ccd963cd01c0553d7f55fd4f"
Oct 30 12:30:09.151771 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.151725    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm_odh-ai-gateway-infra(7e5d41ef-26d9-4de3-886e-a1254ff2291f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f"
Oct 30 12:30:09.158380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.158361    5060 generic.go:356] "Generic (PLEG): container finished" podID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerID="b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7" exitCode=60
Oct 30 12:30:09.158561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.158546    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerDied","Data":"b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7"}
Oct 30 12:30:09.158937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.158923    5060 scope.go:122] "RemoveContainer" containerID="b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7"
Oct 30 12:30:09.159350 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.159332    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl_odh-ai-gateway-infra(3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1"
Oct 30 12:30:09.160288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.160271    5060 scope.go:122] "RemoveContainer" containerID="8c3f47a94a6dff5eea4f2174a2e6835fdc16803ff1b79e6c4d246caf5e9497ac"
Oct 30 12:30:09.161985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.161964    5060 generic.go:356] "Generic (PLEG): container finished" podID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerID="9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25" exitCode=60
Oct 30 12:30:09.162089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.162039    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerDied","Data":"9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25"}
Oct 30 12:30:09.162495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.162329    5060 scope.go:122] "RemoveContainer" containerID="9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25"
Oct 30 12:30:09.162666 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.162646    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s_odh-ai-gateway-infra(d276acb6-2405-47d4-b385-0d293dc2ea38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38"
Oct 30 12:30:09.164076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.164056    5060 generic.go:356] "Generic (PLEG): container finished" podID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerID="0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e" exitCode=60
Oct 30 12:30:09.164177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.164135    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerDied","Data":"0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e"}
Oct 30 12:30:09.164399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.164381    5060 scope.go:122] "RemoveContainer" containerID="0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e"
Oct 30 12:30:09.164695 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.164663    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528_odh-ai-gateway-infra(84f05789-dc11-4f6a-bccc-8d6c1740018e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e"
Oct 30 12:30:09.166122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.166102    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6c1d179-d831-4a74-a377-1d549e622017" containerID="f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96" exitCode=60
Oct 30 12:30:09.166213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.166195    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerDied","Data":"f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96"}
Oct 30 12:30:09.166521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.166500    5060 scope.go:122] "RemoveContainer" containerID="f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96"
Oct 30 12:30:09.166828 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.166804    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9_odh-ai-gateway-infra(a6c1d179-d831-4a74-a377-1d549e622017)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" podUID="a6c1d179-d831-4a74-a377-1d549e622017"
Oct 30 12:30:09.168400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.168377    5060 generic.go:356] "Generic (PLEG): container finished" podID="292ec118-23ed-4030-843d-8a906ff083c7" containerID="b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff" exitCode=60
Oct 30 12:30:09.168533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.168465    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerDied","Data":"b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"}
Oct 30 12:30:09.168721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.168705    5060 scope.go:122] "RemoveContainer" containerID="b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"
Oct 30 12:30:09.168976 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.168950    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7_odh-ai-gateway-infra(292ec118-23ed-4030-843d-8a906ff083c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" podUID="292ec118-23ed-4030-843d-8a906ff083c7"
Oct 30 12:30:09.170494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.170472    5060 generic.go:356] "Generic (PLEG): container finished" podID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerID="13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe" exitCode=60
Oct 30 12:30:09.170586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.170562    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerDied","Data":"13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe"}
Oct 30 12:30:09.170804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.170788    5060 scope.go:122] "RemoveContainer" containerID="13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe"
Oct 30 12:30:09.171045 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.171020    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2_odh-ai-gateway-infra(c89f2ac9-4e23-41f2-a8ae-596ce7000751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751"
Oct 30 12:30:09.172446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.172425    5060 generic.go:356] "Generic (PLEG): container finished" podID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerID="0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b" exitCode=60
Oct 30 12:30:09.172533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.172479    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerDied","Data":"0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b"}
Oct 30 12:30:09.172724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.172710    5060 scope.go:122] "RemoveContainer" containerID="0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b"
Oct 30 12:30:09.172958 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.172937    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92_odh-ai-gateway-infra(63ef289b-15ed-4151-be61-1b5727f8caf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5"
Oct 30 12:30:09.174405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.174383    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerID="e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4" exitCode=60
Oct 30 12:30:09.174519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.174486    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerDied","Data":"e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4"}
Oct 30 12:30:09.174871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.174855    5060 scope.go:122] "RemoveContainer" containerID="e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4"
Oct 30 12:30:09.175088 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:09.175068    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j_odh-ai-gateway-infra(c4438f8b-b45c-4186-b63f-e9226d426ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8"
Oct 30 12:30:09.222505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.210093    5060 scope.go:122] "RemoveContainer" containerID="679d5f8e0f5932bbd5dd7089f14ab0ee286aa37bd7ce635d56ca29c2f633b240"
Oct 30 12:30:09.240118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.240086    5060 scope.go:122] "RemoveContainer" containerID="ffa0229c926f1a47398c8f5819376a22fb927391290b2eb2b3dfc5dc486cb0b2"
Oct 30 12:30:09.262675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.262652    5060 scope.go:122] "RemoveContainer" containerID="48935a5d4eb8250ac06a8bcc1fb0f85897d1d50fb813b2f7baa517deea922259"
Oct 30 12:30:09.323378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.323345    5060 scope.go:122] "RemoveContainer" containerID="3ebbe5bdf4fc2223220de31c581ddbe328dc7f5d198f763bda931011a0d00586"
Oct 30 12:30:09.342520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.342488    5060 scope.go:122] "RemoveContainer" containerID="85558e47f0d18fa419590de77a43f97c66e45ff436fc623dcc6e292eddf01c0b"
Oct 30 12:30:09.360099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.360078    5060 scope.go:122] "RemoveContainer" containerID="6231f5a4ab34caa28c1c0424d96fea190a1d7378401cb71d49dff69e26ee3485"
Oct 30 12:30:09.375857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.375832    5060 scope.go:122] "RemoveContainer" containerID="bfff451c91374aa1041d7c35419aeb308e07258d668c42db385d6f3d33091903"
Oct 30 12:30:09.441907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.441873    5060 scope.go:122] "RemoveContainer" containerID="d012c70b57dcc99d50423c2ef21067c16cee068011cfd1e771ce1e75202af7d9"
Oct 30 12:30:09.467134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.467108    5060 scope.go:122] "RemoveContainer" containerID="5eae8da47a1735efb53be2daf077ed6c00fdb784dbe3cb1a7fe43ec018519d05"
Oct 30 12:30:09.527924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.525908    5060 scope.go:122] "RemoveContainer" containerID="f4ea74a37ce33c2b3e43f28e71eb3526e27eaecc732c8bdde88de05040d84b59"
Oct 30 12:30:09.549672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.549650    5060 scope.go:122] "RemoveContainer" containerID="58f509593837b4798338d1266ff2a1085c855b8173c211b8988bd26dd621374a"
Oct 30 12:30:09.850185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.850138    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main" containerID="cri-o://1b42b5f6e698b2f3a4c0ecfc4f513fa8fbbe1099280f22d79b5f82f4246931f4" gracePeriod=45
Oct 30 12:30:09.932492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:09.930133    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:10.250425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.250394    5060 scope.go:122] "RemoveContainer" containerID="36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488"
Oct 30 12:30:10.250960 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.250929    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr_odh-ai-gateway-infra(0a89196e-f4d8-436e-8bce-144bd3164a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03"
Oct 30 12:30:10.259014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.258989    5060 scope.go:122] "RemoveContainer" containerID="b6a6b8bf2427ec570a7cabf51c436b65ecc0d6d90a7fbdc2610e8651c5a2051a"
Oct 30 12:30:10.259742 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.259718    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z_odh-ai-gateway-infra(a408b674-92df-4821-b568-7f6ee75c5d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" podUID="a408b674-92df-4821-b568-7f6ee75c5d36"
Oct 30 12:30:10.264131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.264103    5060 scope.go:122] "RemoveContainer" containerID="c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364"
Oct 30 12:30:10.264576 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.264546    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8_odh-ai-gateway-infra(4e468f83-2204-4f1d-9a08-f41115e4ef86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86"
Oct 30 12:30:10.267243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.267219    5060 scope.go:122] "RemoveContainer" containerID="222068f2daa46fd328d01ec2f3cd89e0121a4bf7ccd963cd01c0553d7f55fd4f"
Oct 30 12:30:10.267640 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.267611    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm_odh-ai-gateway-infra(7e5d41ef-26d9-4de3-886e-a1254ff2291f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f"
Oct 30 12:30:10.270283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.270263    5060 scope.go:122] "RemoveContainer" containerID="b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7"
Oct 30 12:30:10.270656 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.270632    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl_odh-ai-gateway-infra(3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1"
Oct 30 12:30:10.271436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.271415    5060 scope.go:122] "RemoveContainer" containerID="9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25"
Oct 30 12:30:10.271740 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.271718    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s_odh-ai-gateway-infra(d276acb6-2405-47d4-b385-0d293dc2ea38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38"
Oct 30 12:30:10.274415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.274395    5060 scope.go:122] "RemoveContainer" containerID="0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e"
Oct 30 12:30:10.274715 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.274689    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528_odh-ai-gateway-infra(84f05789-dc11-4f6a-bccc-8d6c1740018e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e"
Oct 30 12:30:10.276344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.276321    5060 scope.go:122] "RemoveContainer" containerID="f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96"
Oct 30 12:30:10.318516 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.276655    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9_odh-ai-gateway-infra(a6c1d179-d831-4a74-a377-1d549e622017)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" podUID="a6c1d179-d831-4a74-a377-1d549e622017"
Oct 30 12:30:10.335028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.334972    5060 scope.go:122] "RemoveContainer" containerID="b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"
Oct 30 12:30:10.335540 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.335506    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7_odh-ai-gateway-infra(292ec118-23ed-4030-843d-8a906ff083c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" podUID="292ec118-23ed-4030-843d-8a906ff083c7"
Oct 30 12:30:10.340174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.340152    5060 scope.go:122] "RemoveContainer" containerID="13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe"
Oct 30 12:30:10.340675 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.340652    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2_odh-ai-gateway-infra(c89f2ac9-4e23-41f2-a8ae-596ce7000751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751"
Oct 30 12:30:10.340867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.340850    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:30:10.343795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.343768    5060 scope.go:122] "RemoveContainer" containerID="0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b"
Oct 30 12:30:10.344058 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.344035    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92_odh-ai-gateway-infra(63ef289b-15ed-4151-be61-1b5727f8caf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5"
Oct 30 12:30:10.345950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.345880    5060 scope.go:122] "RemoveContainer" containerID="e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4"
Oct 30 12:30:10.346209 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.346186    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j_odh-ai-gateway-infra(c4438f8b-b45c-4186-b63f-e9226d426ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8"
Oct 30 12:30:10.347314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.347070    5060 generic.go:356] "Generic (PLEG): container finished" podID="9352f0ab-4592-4083-a373-eb5780165613" containerID="1b42b5f6e698b2f3a4c0ecfc4f513fa8fbbe1099280f22d79b5f82f4246931f4" exitCode=0
Oct 30 12:30:10.347314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.347164    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"
Oct 30 12:30:10.347314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.347236    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" event={"ID":"9352f0ab-4592-4083-a373-eb5780165613","Type":"ContainerDied","Data":"1b42b5f6e698b2f3a4c0ecfc4f513fa8fbbe1099280f22d79b5f82f4246931f4"}
Oct 30 12:30:10.347314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.347260    5060 scope.go:122] "RemoveContainer" containerID="1b42b5f6e698b2f3a4c0ecfc4f513fa8fbbe1099280f22d79b5f82f4246931f4"
Oct 30 12:30:10.349403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.349385    5060 scope.go:122] "RemoveContainer" containerID="89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066"
Oct 30 12:30:10.349647 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:10.349630    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx_odh-ai-gateway-infra(236a39f6-e955-4e84-b437-7592c13f8f76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" podUID="236a39f6-e955-4e84-b437-7592c13f8f76"
Oct 30 12:30:10.423910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.423801    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir\") pod \"9352f0ab-4592-4083-a373-eb5780165613\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") "
Oct 30 12:30:10.423910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.423876    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm\") pod \"9352f0ab-4592-4083-a373-eb5780165613\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") "
Oct 30 12:30:10.424143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.423930    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home\") pod \"9352f0ab-4592-4083-a373-eb5780165613\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") "
Oct 30 12:30:10.424143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.424108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache\") pod \"9352f0ab-4592-4083-a373-eb5780165613\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") "
Oct 30 12:30:10.424252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.424165    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7\" (UniqueName: \"kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7\") pod \"9352f0ab-4592-4083-a373-eb5780165613\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") "
Oct 30 12:30:10.424252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.424203    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs\" (UniqueName: \"kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs\") pod \"9352f0ab-4592-4083-a373-eb5780165613\" (UID: \"9352f0ab-4592-4083-a373-eb5780165613\") "
Oct 30 12:30:10.425987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.424683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir" pod "9352f0ab-4592-4083-a373-eb5780165613" (UID: "9352f0ab-4592-4083-a373-eb5780165613"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:10.425987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.425021    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache" pod "9352f0ab-4592-4083-a373-eb5780165613" (UID: "9352f0ab-4592-4083-a373-eb5780165613"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:10.425987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.425683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home" pod "9352f0ab-4592-4083-a373-eb5780165613" (UID: "9352f0ab-4592-4083-a373-eb5780165613"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:10.426679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.426344    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:10.426679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.426370    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:10.426679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.426384    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:10.428013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.427979    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs" pod "9352f0ab-4592-4083-a373-eb5780165613" (UID: "9352f0ab-4592-4083-a373-eb5780165613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:30:10.429486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.428955    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm" pod "9352f0ab-4592-4083-a373-eb5780165613" (UID: "9352f0ab-4592-4083-a373-eb5780165613"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:10.429908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.429848    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7" pod "9352f0ab-4592-4083-a373-eb5780165613" (UID: "9352f0ab-4592-4083-a373-eb5780165613"). InnerVolumeSpecName "kube-api-access-rndl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:10.528246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.528207    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9352f0ab-4592-4083-a373-eb5780165613-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:10.528246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.528249    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rndl7\" (UniqueName: \"kubernetes.io/projected/9352f0ab-4592-4083-a373-eb5780165613-kube-api-access-rndl7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:10.528551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:10.528266    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9352f0ab-4592-4083-a373-eb5780165613-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:11.056993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:11.056956    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"]
Oct 30 12:30:11.062835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:11.062805    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv"]
Oct 30 12:30:11.324014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:11.323881    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9b2407/test-model-9b2407-kserve-6846698459-4cwpv" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": context deadline exceeded"
Oct 30 12:30:11.836978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:11.836934    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:12.339991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:12.339961    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9352f0ab-4592-4083-a373-eb5780165613" path="/var/lib/kubelet/pods/9352f0ab-4592-4083-a373-eb5780165613/volumes"
Oct 30 12:30:13.679783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:13.679743    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:15.846620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:15.846579    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:17.060750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:17.060708    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:17.211004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:17.210964    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:17.329656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:17.329571    5060 scope.go:122] "RemoveContainer" containerID="39ae35a0a354f18d6891de377e04169be662d6f011e1dfb118c0f0268caf2fa2"
Oct 30 12:30:17.669213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:17.669166    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:18.328056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.328021    5060 scope.go:122] "RemoveContainer" containerID="6bd63bea1ae3d741bc674f23dc56370570710fcbb1f6e1893cc034882158fe9c"
Oct 30 12:30:18.330840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.328630    5060 scope.go:122] "RemoveContainer" containerID="2cda6514c7f099ef1d81bd08bcd78191815ff8308beedb4d3c2a7aa01d8d8f82"
Oct 30 12:30:18.330840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.328818    5060 scope.go:122] "RemoveContainer" containerID="08e99d7dfc7bfbb224abc83f44bf2e046abbbfdad517448aeaf471ea29d8c8e2"
Oct 30 12:30:18.554158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.554128    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerStarted","Data":"d395aeac6f880111b17ffe87d282c816bfa2d6a198f6a15636c878b3222beb10"}
Oct 30 12:30:18.556780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.556753    5060 generic.go:356] "Generic (PLEG): container finished" podID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerID="ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5" exitCode=0
Oct 30 12:30:18.557017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.556900    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" event={"ID":"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a","Type":"ContainerDied","Data":"ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"}
Oct 30 12:30:18.557517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.557502    5060 scope.go:122] "RemoveContainer" containerID="ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"
Oct 30 12:30:18.560763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.560727    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:30:18.560861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.560820    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:30:18.560908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:18.560867    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:30:19.037067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.037032    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"]
Oct 30 12:30:19.044472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.044424    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a317845b-xmhc8"]
Oct 30 12:30:19.205479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.200573    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:30:19.328515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.327970    5060 scope.go:122] "RemoveContainer" containerID="b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7"
Oct 30 12:30:19.328515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.328171    5060 scope.go:122] "RemoveContainer" containerID="89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066"
Oct 30 12:30:19.329576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.329537    5060 scope.go:122] "RemoveContainer" containerID="4f3098c99e36c769a525ec21f3543e2d3d57954d2d816cc7849a3f9eb8648257"
Oct 30 12:30:19.330392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.330067    5060 scope.go:122] "RemoveContainer" containerID="9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25"
Oct 30 12:30:19.371226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.371069    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:19.575485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.574607    5060 generic.go:356] "Generic (PLEG): container finished" podID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerID="d395aeac6f880111b17ffe87d282c816bfa2d6a198f6a15636c878b3222beb10" exitCode=60
Oct 30 12:30:19.575485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.574804    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerDied","Data":"d395aeac6f880111b17ffe87d282c816bfa2d6a198f6a15636c878b3222beb10"}
Oct 30 12:30:19.575485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.574859    5060 scope.go:122] "RemoveContainer" containerID="39ae35a0a354f18d6891de377e04169be662d6f011e1dfb118c0f0268caf2fa2"
Oct 30 12:30:19.575485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.575353    5060 scope.go:122] "RemoveContainer" containerID="d395aeac6f880111b17ffe87d282c816bfa2d6a198f6a15636c878b3222beb10"
Oct 30 12:30:19.579478 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:19.575822    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr_odh-ai-gateway-infra(e03a2654-376e-4fb8-a9fd-2b5bf73e3f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11"
Oct 30 12:30:19.648578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.648546    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"]
Oct 30 12:30:19.657967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.657923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerStarted","Data":"7cc9804a31a05baa92cae487cb5f2c91ade3fd502cb56064c62f573dc2e9bbea"}
Oct 30 12:30:19.664292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.664181    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerStarted","Data":"8a5f41eb0139717a306bcae3a78db1d8904a793d26aa2e922a524808332009d6"}
Oct 30 12:30:19.667092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.667054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" event={"ID":"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a","Type":"ContainerStarted","Data":"a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1"}
Oct 30 12:30:19.676316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.670303    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerStarted","Data":"26bc9aad33d1fc0a76c4c7e42fb13a648351875187b76abbc5c6eb1424bc0f1c"}
Oct 30 12:30:19.996543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.996507    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:30:19.997422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.997391    5060 patch_prober.go:29] interesting pod/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body=
Oct 30 12:30:19.999500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:19.999466    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused"
Oct 30 12:30:20.046743 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:20.032146    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:30:20.342091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.342060    5060 scope.go:122] "RemoveContainer" containerID="0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e"
Oct 30 12:30:20.344116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.343585    5060 scope.go:122] "RemoveContainer" containerID="13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe"
Oct 30 12:30:20.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.344239    5060 scope.go:122] "RemoveContainer" containerID="a22a97668c61911043e46e59d411dafb0340a5b184bd3afeb9802a7937cdc8d2"
Oct 30 12:30:20.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.344338    5060 scope.go:122] "RemoveContainer" containerID="11c743e8464c3b001d30257d0c8580159e48d1eb40d66ac46a66f2f9aa5e8de2"
Oct 30 12:30:20.344968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.344922    5060 scope.go:122] "RemoveContainer" containerID="b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"
Oct 30 12:30:20.347229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.345706    5060 scope.go:122] "RemoveContainer" containerID="5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767"
Oct 30 12:30:20.360038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.360009    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a249c5cb-3cf6-43d7-bd2d-e42882917ae1" path="/var/lib/kubelet/pods/a249c5cb-3cf6-43d7-bd2d-e42882917ae1/volumes"
Oct 30 12:30:20.752539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.748445    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"]
Oct 30 12:30:20.752539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.749916    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main"
Oct 30 12:30:20.752539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.749951    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main"
Oct 30 12:30:20.752539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.750252    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9352f0ab-4592-4083-a373-eb5780165613" containerName="main"
Oct 30 12:30:20.759379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.759358    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.761298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.761273    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"]
Oct 30 12:30:20.762433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.762279    5060 generic.go:356] "Generic (PLEG): container finished" podID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerID="26bc9aad33d1fc0a76c4c7e42fb13a648351875187b76abbc5c6eb1424bc0f1c" exitCode=60
Oct 30 12:30:20.762433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.762349    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerDied","Data":"26bc9aad33d1fc0a76c4c7e42fb13a648351875187b76abbc5c6eb1424bc0f1c"}
Oct 30 12:30:20.762433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.762409    5060 scope.go:122] "RemoveContainer" containerID="08e99d7dfc7bfbb224abc83f44bf2e046abbbfdad517448aeaf471ea29d8c8e2"
Oct 30 12:30:20.765523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.763365    5060 scope.go:122] "RemoveContainer" containerID="26bc9aad33d1fc0a76c4c7e42fb13a648351875187b76abbc5c6eb1424bc0f1c"
Oct 30 12:30:20.765523 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:20.763770    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4_odh-ai-gateway-infra(3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096"
Oct 30 12:30:20.772668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.772633    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerStarted","Data":"2abd280e4c5da20280c4d56fbe68ce0effc6f774ecfe548305ea51e83a8dcadc"}
Oct 30 12:30:20.775747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.775711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerStarted","Data":"355826bb7c509cf18fa5c19c26ba1a6e4c39a019e24cb61b0f90e484939438c3"}
Oct 30 12:30:20.798395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.798359    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0190d8\"/\"default-dockercfg-kghzl\""
Oct 30 12:30:20.798558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.798491    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0190d8\"/\"openshift-service-ca.crt\""
Oct 30 12:30:20.798681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.798635    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0190d8\"/\"test-model-0190d8-kserve-self-signed-certs\""
Oct 30 12:30:20.798752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.798704    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0190d8\"/\"kube-root-ca.crt\""
Oct 30 12:30:20.852919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.852881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:20.855786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.855759    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.856579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.856560    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.856807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.856790    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.857115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.857091    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.857993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.857979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsch\" (UniqueName: \"kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.858164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.858154    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.875962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.868223    5060 generic.go:356] "Generic (PLEG): container finished" podID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerID="7cc9804a31a05baa92cae487cb5f2c91ade3fd502cb56064c62f573dc2e9bbea" exitCode=60
Oct 30 12:30:20.875962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.868403    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerDied","Data":"7cc9804a31a05baa92cae487cb5f2c91ade3fd502cb56064c62f573dc2e9bbea"}
Oct 30 12:30:20.875962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.873945    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerStarted","Data":"dd0513fef3714d39f4945a63d1f01c7a0eda3e1c0982c7f42ec12bbfbb6caac2"}
Oct 30 12:30:20.961354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961322    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.961560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961371    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.961560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961391    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.961560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961487    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.961560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961560    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsch\" (UniqueName: \"kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.961841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961604    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.961981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.961960    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.962144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.962126    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.962306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.962277    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:20.964531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.964500    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerStarted","Data":"cb310af7271e4f32e440a53f0214a1ee321c27551fce31fbe4445efc7359eafc"}
Oct 30 12:30:20.965785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:20.965731    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:21.017815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.015152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:21.035565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.031740    5060 patch_prober.go:29] interesting pod/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body=
Oct 30 12:30:21.035565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.031801    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused"
Oct 30 12:30:21.076485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.055924    5060 generic.go:356] "Generic (PLEG): container finished" podID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerID="8a5f41eb0139717a306bcae3a78db1d8904a793d26aa2e922a524808332009d6" exitCode=60
Oct 30 12:30:21.076485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.059682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerDied","Data":"8a5f41eb0139717a306bcae3a78db1d8904a793d26aa2e922a524808332009d6"}
Oct 30 12:30:21.198990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.191228    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:21.198990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.192531    5060 scope.go:122] "RemoveContainer" containerID="8a5f41eb0139717a306bcae3a78db1d8904a793d26aa2e922a524808332009d6"
Oct 30 12:30:21.198990 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:21.192938    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22_odh-ai-gateway-infra(f0a6acb4-68df-4d60-add2-101fe1ddf61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e"
Oct 30 12:30:21.335085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.271900    5060 scope.go:122] "RemoveContainer" containerID="6bd63bea1ae3d741bc674f23dc56370570710fcbb1f6e1893cc034882158fe9c"
Oct 30 12:30:21.335085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.315573    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"]
Oct 30 12:30:21.348514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.345772    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsch\" (UniqueName: \"kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch\") pod \"test-model-0190d8-kserve-c5d874c47-zbfpr\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") " pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:21.421479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.390418    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"]
Oct 30 12:30:21.441997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.437042    5060 scope.go:122] "RemoveContainer" containerID="e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4"
Oct 30 12:30:21.441997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.437310    5060 scope.go:122] "RemoveContainer" containerID="0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b"
Oct 30 12:30:21.450232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.449295    5060 scope.go:122] "RemoveContainer" containerID="9f1a3dd280bb062d6a313b7005c9476ddd3ea65c28c1f16098aa75f23e22d2fb"
Oct 30 12:30:21.450232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.449693    5060 scope.go:122] "RemoveContainer" containerID="031446c081f23c7f2f4f97e5d6930b5a840e9b18f9385accfa45c980136eef23"
Oct 30 12:30:21.450232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.449889    5060 scope.go:122] "RemoveContainer" containerID="b6a6b8bf2427ec570a7cabf51c436b65ecc0d6d90a7fbdc2610e8651c5a2051a"
Oct 30 12:30:21.452077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.451724    5060 scope.go:122] "RemoveContainer" containerID="36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488"
Oct 30 12:30:21.452954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.452938    5060 scope.go:122] "RemoveContainer" containerID="c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364"
Oct 30 12:30:21.546053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.461550    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:21.555741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.554444    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"]
Oct 30 12:30:21.734205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.734138    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" podStartSLOduration=21.734117853 podStartE2EDuration="21.734117853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:21.058984673 +0000 UTC m=+3624.853869179" watchObservedRunningTime="2026-10-30 12:30:21.734117853 +0000 UTC m=+3625.529002349"
Oct 30 12:30:21.946123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.945819    5060 scope.go:122] "RemoveContainer" containerID="2cda6514c7f099ef1d81bd08bcd78191815ff8308beedb4d3c2a7aa01d8d8f82"
Oct 30 12:30:21.961735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:21.961698    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"]
Oct 30 12:30:22.035556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.035432    5060 patch_prober.go:29] interesting pod/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body=
Oct 30 12:30:22.035556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.035509    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused"
Oct 30 12:30:22.063842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.063756    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"]
Oct 30 12:30:22.066824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.066800    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:22.141782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.141744    5060 generic.go:356] "Generic (PLEG): container finished" podID="236a39f6-e955-4e84-b437-7592c13f8f76" containerID="355826bb7c509cf18fa5c19c26ba1a6e4c39a019e24cb61b0f90e484939438c3" exitCode=60
Oct 30 12:30:22.141948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.141854    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerDied","Data":"355826bb7c509cf18fa5c19c26ba1a6e4c39a019e24cb61b0f90e484939438c3"}
Oct 30 12:30:22.141948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.141912    5060 scope.go:122] "RemoveContainer" containerID="89d86263a9422c90f30f46b054cd7913b20983f65226e925efeca66332f7e066"
Oct 30 12:30:22.147060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.147015    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr" event={"ID":"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11","Type":"ContainerDied","Data":"8404ac62ae5e632cf3ae035f83afbde54bc30da80cf5cfd4976999687a5bdbfa"}
Oct 30 12:30:22.147266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.147247    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8404ac62ae5e632cf3ae035f83afbde54bc30da80cf5cfd4976999687a5bdbfa"
Oct 30 12:30:22.149498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.149472    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerStarted","Data":"d8438507c54ec735e9b261ac5e3c32076b2090af5f16bd36e6b4f91505c13267"}
Oct 30 12:30:22.150873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.150852    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"
Oct 30 12:30:22.150987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.150918    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb" event={"ID":"144e4de0-8203-4e08-a384-f6a89b6627a5","Type":"ContainerDied","Data":"80bfe18a35f29f341c1beb90bb388ee2ee805791ebd7ad0474f4383ad3887dbb"}
Oct 30 12:30:22.152813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.152790    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerStarted","Data":"335df6cf1c7048690b6da945c12f4751eab7780a84d1e577eb0a406c5a30e8b6"}
Oct 30 12:30:22.154581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.154565    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6\" (UniqueName: \"kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6\") pod \"144e4de0-8203-4e08-a384-f6a89b6627a5\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") "
Oct 30 12:30:22.154810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.154796    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca\") pod \"144e4de0-8203-4e08-a384-f6a89b6627a5\" (UID: \"144e4de0-8203-4e08-a384-f6a89b6627a5\") "
Oct 30 12:30:22.155124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.155104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca" pod "144e4de0-8203-4e08-a384-f6a89b6627a5" (UID: "144e4de0-8203-4e08-a384-f6a89b6627a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:22.155677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.155650    5060 generic.go:356] "Generic (PLEG): container finished" podID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerID="dd0513fef3714d39f4945a63d1f01c7a0eda3e1c0982c7f42ec12bbfbb6caac2" exitCode=60
Oct 30 12:30:22.155824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.155724    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerDied","Data":"dd0513fef3714d39f4945a63d1f01c7a0eda3e1c0982c7f42ec12bbfbb6caac2"}
Oct 30 12:30:22.156028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.156014    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144e4de0-8203-4e08-a384-f6a89b6627a5-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:22.156110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.156016    5060 scope.go:122] "RemoveContainer" containerID="dd0513fef3714d39f4945a63d1f01c7a0eda3e1c0982c7f42ec12bbfbb6caac2"
Oct 30 12:30:22.156415 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:22.156397    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl_odh-ai-gateway-infra(3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1"
Oct 30 12:30:22.157934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.157787    5060 generic.go:356] "Generic (PLEG): container finished" podID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerID="cb310af7271e4f32e440a53f0214a1ee321c27551fce31fbe4445efc7359eafc" exitCode=60
Oct 30 12:30:22.157934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.157852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerDied","Data":"cb310af7271e4f32e440a53f0214a1ee321c27551fce31fbe4445efc7359eafc"}
Oct 30 12:30:22.158328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.158312    5060 scope.go:122] "RemoveContainer" containerID="cb310af7271e4f32e440a53f0214a1ee321c27551fce31fbe4445efc7359eafc"
Oct 30 12:30:22.158615 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:22.158597    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s_odh-ai-gateway-infra(d276acb6-2405-47d4-b385-0d293dc2ea38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38"
Oct 30 12:30:22.159961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.159943    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerStarted","Data":"11fc25f7a9971b0f45e4d4af1bf4d1ce633579e93fdcf400278178b0fb84a271"}
Oct 30 12:30:22.163986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.163963    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerStarted","Data":"6d760ad873e846b418f3ee1d55a0df513100d5f1f2ca3bdbd675e38e59767f94"}
Oct 30 12:30:22.166664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.166643    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerStarted","Data":"d27b4b9dec0b256c8e5f9411c09569c1443ce0e87f56656deaa270ad8bf32baa"}
Oct 30 12:30:22.170893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.170872    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerStarted","Data":"821d7c57fa0648c0bcac593586bb1b17ae9fc2a743093c5f9fc11ec4d32f9fdc"}
Oct 30 12:30:22.172612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.172582    5060 generic.go:356] "Generic (PLEG): container finished" podID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerID="2abd280e4c5da20280c4d56fbe68ce0effc6f774ecfe548305ea51e83a8dcadc" exitCode=60
Oct 30 12:30:22.172717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.172642    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerDied","Data":"2abd280e4c5da20280c4d56fbe68ce0effc6f774ecfe548305ea51e83a8dcadc"}
Oct 30 12:30:22.238328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.238283    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6" pod "144e4de0-8203-4e08-a384-f6a89b6627a5" (UID: "144e4de0-8203-4e08-a384-f6a89b6627a5"). InnerVolumeSpecName "kube-api-access-56fh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:22.265108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.265059    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fh6\" (UniqueName: \"kubernetes.io/projected/144e4de0-8203-4e08-a384-f6a89b6627a5-kube-api-access-56fh6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:22.333894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.330610    5060 scope.go:122] "RemoveContainer" containerID="f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96"
Oct 30 12:30:22.367941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.367546    5060 scope.go:122] "RemoveContainer" containerID="7cc9804a31a05baa92cae487cb5f2c91ade3fd502cb56064c62f573dc2e9bbea"
Oct 30 12:30:22.373839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.373559    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"]
Oct 30 12:30:22.392897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.389637    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:22.472440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.467237    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca\") pod \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") "
Oct 30 12:30:22.472440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.467917    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca" pod "e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" (UID: "e03a2654-376e-4fb8-a9fd-2b5bf73e3f11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:22.472440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.468285    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx\" (UniqueName: \"kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx\") pod \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\" (UID: \"e03a2654-376e-4fb8-a9fd-2b5bf73e3f11\") "
Oct 30 12:30:22.472440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.469282    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:22.521597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.521565    5060 scope.go:122] "RemoveContainer" containerID="b38c94432559ccf041547efea36dab5f58d99406c6fddb72eaf9661a338c03b7"
Oct 30 12:30:22.532470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.531917    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"]
Oct 30 12:30:22.548497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.548464    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"]
Oct 30 12:30:22.573402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.573365    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx" pod "e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" (UID: "e03a2654-376e-4fb8-a9fd-2b5bf73e3f11"). InnerVolumeSpecName "kube-api-access-vdslx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:22.658284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.657272    5060 scope.go:122] "RemoveContainer" containerID="9eaf522eeff2af8ca909d945e5088d49c29aa133e31a0c8d2ad4c665a9b2cd25"
Oct 30 12:30:22.675494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:22.673283    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdslx\" (UniqueName: \"kubernetes.io/projected/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11-kube-api-access-vdslx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.002199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.002163    5060 patch_prober.go:29] interesting pod/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body=
Oct 30 12:30:23.002510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.002480    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused"
Oct 30 12:30:23.012252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.012218    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"]
Oct 30 12:30:23.032736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.032705    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bae155-29889390-c2qtb"]
Oct 30 12:30:23.039380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.039353    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:23.147412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.147386    5060 scope.go:122] "RemoveContainer" containerID="4f3098c99e36c769a525ec21f3543e2d3d57954d2d816cc7849a3f9eb8648257"
Oct 30 12:30:23.170778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.170748    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:23.232826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.231580    5060 generic.go:356] "Generic (PLEG): container finished" podID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerID="d27b4b9dec0b256c8e5f9411c09569c1443ce0e87f56656deaa270ad8bf32baa" exitCode=60
Oct 30 12:30:23.232826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.231686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerDied","Data":"d27b4b9dec0b256c8e5f9411c09569c1443ce0e87f56656deaa270ad8bf32baa"}
Oct 30 12:30:23.232826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.231774    5060 scope.go:122] "RemoveContainer" containerID="13088feb01ddad51219c803368e6d897ee98fcbbaa7038ca8a29d27f54211ebe"
Oct 30 12:30:23.232826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.232200    5060 scope.go:122] "RemoveContainer" containerID="d27b4b9dec0b256c8e5f9411c09569c1443ce0e87f56656deaa270ad8bf32baa"
Oct 30 12:30:23.232826 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:23.232603    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2_odh-ai-gateway-infra(c89f2ac9-4e23-41f2-a8ae-596ce7000751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751"
Oct 30 12:30:23.237897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.237874    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:23.241554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.240882    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerStarted","Data":"50927005db74351fefb143387f5ef7cfe13d86cef72519294ed7556a7fe7aebe"}
Oct 30 12:30:23.242085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.242065    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:23.243075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.243025    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"
Oct 30 12:30:23.243075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.243069    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4" event={"ID":"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096","Type":"ContainerDied","Data":"17dc4a38515d54a0bcfa7f8081683c12f82464b3dc2ce22d721ef43fe52d6758"}
Oct 30 12:30:23.246109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.246082    5060 generic.go:356] "Generic (PLEG): container finished" podID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerID="821d7c57fa0648c0bcac593586bb1b17ae9fc2a743093c5f9fc11ec4d32f9fdc" exitCode=60
Oct 30 12:30:23.246224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.246168    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerDied","Data":"821d7c57fa0648c0bcac593586bb1b17ae9fc2a743093c5f9fc11ec4d32f9fdc"}
Oct 30 12:30:23.246590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.246567    5060 scope.go:122] "RemoveContainer" containerID="821d7c57fa0648c0bcac593586bb1b17ae9fc2a743093c5f9fc11ec4d32f9fdc"
Oct 30 12:30:23.247809 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:23.247779    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp_odh-ai-gateway-infra(4a830ea6-1b84-4ddb-abe8-01830a9ce9e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3"
Oct 30 12:30:23.255361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.255339    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"
Oct 30 12:30:23.255543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.255520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7" event={"ID":"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8","Type":"ContainerDied","Data":"d495b3c779e9df7e8b2fcb1d44eb3af57ee6862ce4ae8ad9042cce89e3a2bd6d"}
Oct 30 12:30:23.257119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.257100    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"
Oct 30 12:30:23.257729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.257138    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx" event={"ID":"236a39f6-e955-4e84-b437-7592c13f8f76","Type":"ContainerDied","Data":"a9c0bfcf11955ee8d37f395e4953e93e6b757e90f579d39f1538b0bd0fb9a972"}
Oct 30 12:30:23.258933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.258915    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:23.259779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.259761    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerStarted","Data":"85322989ca7ea21d4b02b82b7eddd24001912ba6aa3d2e8bd83c39e74ad86d67"}
Oct 30 12:30:23.260857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.260838    5060 scope.go:122] "RemoveContainer" containerID="26bc9aad33d1fc0a76c4c7e42fb13a648351875187b76abbc5c6eb1424bc0f1c"
Oct 30 12:30:23.261798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.261774    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"]
Oct 30 12:30:23.263059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.263039    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr" event={"ID":"cf525b07-aa77-49bb-aec4-dafb8d5b3a27","Type":"ContainerStarted","Data":"4fad2271a70861f6b0e2604210626a82d82e7c20b082d9e0cd18526cc1fc7723"}
Oct 30 12:30:23.266107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.266080    5060 generic.go:356] "Generic (PLEG): container finished" podID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerID="d8438507c54ec735e9b261ac5e3c32076b2090af5f16bd36e6b4f91505c13267" exitCode=60
Oct 30 12:30:23.266276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.266152    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerDied","Data":"d8438507c54ec735e9b261ac5e3c32076b2090af5f16bd36e6b4f91505c13267"}
Oct 30 12:30:23.266864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.266846    5060 scope.go:122] "RemoveContainer" containerID="d8438507c54ec735e9b261ac5e3c32076b2090af5f16bd36e6b4f91505c13267"
Oct 30 12:30:23.267564 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:23.267542    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc_odh-ai-gateway-infra(4d41c30a-79d8-463b-94a6-4da554dbf161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161"
Oct 30 12:30:23.274499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.274474    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerStarted","Data":"bb82036947457e663de68a362f62561dc269fcea8c9b21bd78afc89be8cd6ceb"}
Oct 30 12:30:23.332482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.332096    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca\") pod \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") "
Oct 30 12:30:23.332762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.332741    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm\" (UniqueName: \"kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm\") pod \"236a39f6-e955-4e84-b437-7592c13f8f76\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") "
Oct 30 12:30:23.332925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.332889    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp\" (UniqueName: \"kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp\") pod \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") "
Oct 30 12:30:23.333090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.333076    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca\") pod \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\" (UID: \"f0a6acb4-68df-4d60-add2-101fe1ddf61e\") "
Oct 30 12:30:23.333198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.333186    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8\" (UniqueName: \"kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8\") pod \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") "
Oct 30 12:30:23.333307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.333295    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca\") pod \"236a39f6-e955-4e84-b437-7592c13f8f76\" (UID: \"236a39f6-e955-4e84-b437-7592c13f8f76\") "
Oct 30 12:30:23.333725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.333711    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg\" (UniqueName: \"kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg\") pod \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\" (UID: \"0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8\") "
Oct 30 12:30:23.333861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.333852    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca\") pod \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\" (UID: \"3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096\") "
Oct 30 12:30:23.334699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.334668    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca" pod "0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" (UID: "0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:23.334915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.334867    5060 generic.go:356] "Generic (PLEG): container finished" podID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerID="335df6cf1c7048690b6da945c12f4751eab7780a84d1e577eb0a406c5a30e8b6" exitCode=60
Oct 30 12:30:23.335061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.334967    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerDied","Data":"335df6cf1c7048690b6da945c12f4751eab7780a84d1e577eb0a406c5a30e8b6"}
Oct 30 12:30:23.335376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.335351    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.335652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.332377    5060 scope.go:122] "RemoveContainer" containerID="222068f2daa46fd328d01ec2f3cd89e0121a4bf7ccd963cd01c0553d7f55fd4f"
Oct 30 12:30:23.337659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.337120    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca" pod "f0a6acb4-68df-4d60-add2-101fe1ddf61e" (UID: "f0a6acb4-68df-4d60-add2-101fe1ddf61e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:23.337659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.337391    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca" pod "236a39f6-e955-4e84-b437-7592c13f8f76" (UID: "236a39f6-e955-4e84-b437-7592c13f8f76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:23.337659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.337532    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca" pod "3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" (UID: "3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:23.338064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.337940    5060 scope.go:122] "RemoveContainer" containerID="a22a97668c61911043e46e59d411dafb0340a5b184bd3afeb9802a7937cdc8d2"
Oct 30 12:30:23.338064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.338047    5060 scope.go:122] "RemoveContainer" containerID="335df6cf1c7048690b6da945c12f4751eab7780a84d1e577eb0a406c5a30e8b6"
Oct 30 12:30:23.338774 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:23.338343    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2_odh-ai-gateway-infra(e3ba43b6-963c-4f0e-ab47-639eebd7497f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f"
Oct 30 12:30:23.339175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.339139    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8" pod "3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" (UID: "3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096"). InnerVolumeSpecName "kube-api-access-wdzp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:23.339839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.339814    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg" pod "0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" (UID: "0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8"). InnerVolumeSpecName "kube-api-access-7mvzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:23.340410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.340376    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp" pod "f0a6acb4-68df-4d60-add2-101fe1ddf61e" (UID: "f0a6acb4-68df-4d60-add2-101fe1ddf61e"). InnerVolumeSpecName "kube-api-access-9gvfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:23.342035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.342001    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm" pod "236a39f6-e955-4e84-b437-7592c13f8f76" (UID: "236a39f6-e955-4e84-b437-7592c13f8f76"). InnerVolumeSpecName "kube-api-access-z6nmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:23.363521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.362699    5060 generic.go:356] "Generic (PLEG): container finished" podID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerID="11fc25f7a9971b0f45e4d4af1bf4d1ce633579e93fdcf400278178b0fb84a271" exitCode=60
Oct 30 12:30:23.363521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.362867    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerDied","Data":"11fc25f7a9971b0f45e4d4af1bf4d1ce633579e93fdcf400278178b0fb84a271"}
Oct 30 12:30:23.363521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.363273    5060 scope.go:122] "RemoveContainer" containerID="11fc25f7a9971b0f45e4d4af1bf4d1ce633579e93fdcf400278178b0fb84a271"
Oct 30 12:30:23.365122 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:23.364041    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528_odh-ai-gateway-infra(84f05789-dc11-4f6a-bccc-8d6c1740018e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e"
Oct 30 12:30:23.373526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.367145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22" event={"ID":"f0a6acb4-68df-4d60-add2-101fe1ddf61e","Type":"ContainerDied","Data":"928deb9e17c0677048bbb915ad906fb77897b1309930df0552ffdd3c666264d2"}
Oct 30 12:30:23.373526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.367252    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"
Oct 30 12:30:23.373526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.371326    5060 generic.go:356] "Generic (PLEG): container finished" podID="292ec118-23ed-4030-843d-8a906ff083c7" containerID="6d760ad873e846b418f3ee1d55a0df513100d5f1f2ca3bdbd675e38e59767f94" exitCode=60
Oct 30 12:30:23.373526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.371467    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"
Oct 30 12:30:23.373526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.372983    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerDied","Data":"6d760ad873e846b418f3ee1d55a0df513100d5f1f2ca3bdbd675e38e59767f94"}
Oct 30 12:30:23.373526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.373521    5060 scope.go:122] "RemoveContainer" containerID="6d760ad873e846b418f3ee1d55a0df513100d5f1f2ca3bdbd675e38e59767f94"
Oct 30 12:30:23.374283 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:23.373904    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7_odh-ai-gateway-infra(292ec118-23ed-4030-843d-8a906ff083c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" podUID="292ec118-23ed-4030-843d-8a906ff083c7"
Oct 30 12:30:23.434131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.431856    5060 scope.go:122] "RemoveContainer" containerID="2abd280e4c5da20280c4d56fbe68ce0effc6f774ecfe548305ea51e83a8dcadc"
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437666    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6acb4-68df-4d60-add2-101fe1ddf61e-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437698    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdzp8\" (UniqueName: \"kubernetes.io/projected/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-kube-api-access-wdzp8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437718    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236a39f6-e955-4e84-b437-7592c13f8f76-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437733    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvzg\" (UniqueName: \"kubernetes.io/projected/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8-kube-api-access-7mvzg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437748    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437765    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6nmm\" (UniqueName: \"kubernetes.io/projected/236a39f6-e955-4e84-b437-7592c13f8f76-kube-api-access-z6nmm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.441593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.437780    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gvfp\" (UniqueName: \"kubernetes.io/projected/f0a6acb4-68df-4d60-add2-101fe1ddf61e-kube-api-access-9gvfp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:23.475300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.474418    5060 scope.go:122] "RemoveContainer" containerID="355826bb7c509cf18fa5c19c26ba1a6e4c39a019e24cb61b0f90e484939438c3"
Oct 30 12:30:23.555356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.555326    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"]
Oct 30 12:30:23.634601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.634468    5060 scope.go:122] "RemoveContainer" containerID="5f68704fa99d5cc155677bf3ac829ddd476be8efa9cd001a08c2075069585767"
Oct 30 12:30:23.658164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.658127    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"]
Oct 30 12:30:23.743706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.743675    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"]
Oct 30 12:30:23.762770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.762624    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"]
Oct 30 12:30:23.838535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.836401    5060 scope.go:122] "RemoveContainer" containerID="11c743e8464c3b001d30257d0c8580159e48d1eb40d66ac46a66f2f9aa5e8de2"
Oct 30 12:30:23.863967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:23.863933    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"]
Oct 30 12:30:24.024053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.024018    5060 patch_prober.go:29] interesting pod/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body=
Oct 30 12:30:24.024902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.024862    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused"
Oct 30 12:30:24.242357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.242322    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"]
Oct 30 12:30:24.258205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.258168    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b15ec3-29889390-lv4m4"]
Oct 30 12:30:24.265017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.264939    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"]
Oct 30 12:30:24.271618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.271591    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889390-cmmn7"]
Oct 30 12:30:24.341181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.341150    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" path="/var/lib/kubelet/pods/0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8/volumes"
Oct 30 12:30:24.341932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.341913    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" path="/var/lib/kubelet/pods/144e4de0-8203-4e08-a384-f6a89b6627a5/volumes"
Oct 30 12:30:24.342571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.342554    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" path="/var/lib/kubelet/pods/3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096/volumes"
Oct 30 12:30:24.343468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.343436    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"]
Oct 30 12:30:24.346330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.346309    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889390-wkd22"]
Oct 30 12:30:24.375486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.375379    5060 scope.go:122] "RemoveContainer" containerID="0475d9d5b7b12312d7949a83744545cfc65a228973b455d8068249de4e1f6d5e"
Oct 30 12:30:24.419747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.419660    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:24.422383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.422354    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:24.428499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.425688    5060 generic.go:356] "Generic (PLEG): container finished" podID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerID="85322989ca7ea21d4b02b82b7eddd24001912ba6aa3d2e8bd83c39e74ad86d67" exitCode=60
Oct 30 12:30:24.428499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.425752    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerDied","Data":"85322989ca7ea21d4b02b82b7eddd24001912ba6aa3d2e8bd83c39e74ad86d67"}
Oct 30 12:30:24.428499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.426188    5060 scope.go:122] "RemoveContainer" containerID="85322989ca7ea21d4b02b82b7eddd24001912ba6aa3d2e8bd83c39e74ad86d67"
Oct 30 12:30:24.428499 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.426638    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr_odh-ai-gateway-infra(0a89196e-f4d8-436e-8bce-144bd3164a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03"
Oct 30 12:30:24.428920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.428891    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr" event={"ID":"cf525b07-aa77-49bb-aec4-dafb8d5b3a27","Type":"ContainerStarted","Data":"880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e"}
Oct 30 12:30:24.433983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.433962    5060 scope.go:122] "RemoveContainer" containerID="8a5f41eb0139717a306bcae3a78db1d8904a793d26aa2e922a524808332009d6"
Oct 30 12:30:24.436335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.435701    5060 generic.go:356] "Generic (PLEG): container finished" podID="a408b674-92df-4821-b568-7f6ee75c5d36" containerID="8ea9cd69b9b5823abcbb855628a9cb7789396499a1c2caa894db804580e625db" exitCode=60
Oct 30 12:30:24.436335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.435778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" event={"ID":"a408b674-92df-4821-b568-7f6ee75c5d36","Type":"ContainerDied","Data":"8ea9cd69b9b5823abcbb855628a9cb7789396499a1c2caa894db804580e625db"}
Oct 30 12:30:24.436335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.436152    5060 scope.go:122] "RemoveContainer" containerID="8ea9cd69b9b5823abcbb855628a9cb7789396499a1c2caa894db804580e625db"
Oct 30 12:30:24.436957 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.436927    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z_odh-ai-gateway-infra(a408b674-92df-4821-b568-7f6ee75c5d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" podUID="a408b674-92df-4821-b568-7f6ee75c5d36"
Oct 30 12:30:24.441824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.441800    5060 generic.go:356] "Generic (PLEG): container finished" podID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerID="bb82036947457e663de68a362f62561dc269fcea8c9b21bd78afc89be8cd6ceb" exitCode=60
Oct 30 12:30:24.441951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.441876    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerDied","Data":"bb82036947457e663de68a362f62561dc269fcea8c9b21bd78afc89be8cd6ceb"}
Oct 30 12:30:24.442237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.442219    5060 scope.go:122] "RemoveContainer" containerID="bb82036947457e663de68a362f62561dc269fcea8c9b21bd78afc89be8cd6ceb"
Oct 30 12:30:24.442606 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.442583    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8_odh-ai-gateway-infra(4e468f83-2204-4f1d-9a08-f41115e4ef86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86"
Oct 30 12:30:24.445154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.445127    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerStarted","Data":"d2d35aef28dea0db50c977472cb78eae0fa12264f71830c54593c52397cb4f7f"}
Oct 30 12:30:24.449628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.449603    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"]
Oct 30 12:30:24.451312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.451168    5060 scope.go:122] "RemoveContainer" containerID="b5a1b23cee097d9c83c41247feb4983af3d86e2f24bc39fd5831edc15ab008ff"
Oct 30 12:30:24.453069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.453042    5060 generic.go:356] "Generic (PLEG): container finished" podID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerID="f0887ecd2900b945448d4680b46dd0acd1149f02718d52295d91808e45e87403" exitCode=60
Oct 30 12:30:24.453277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.453126    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" event={"ID":"d59ed183-da6d-4132-a925-3a23af0aeb20","Type":"ContainerDied","Data":"f0887ecd2900b945448d4680b46dd0acd1149f02718d52295d91808e45e87403"}
Oct 30 12:30:24.453643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.453622    5060 scope.go:122] "RemoveContainer" containerID="f0887ecd2900b945448d4680b46dd0acd1149f02718d52295d91808e45e87403"
Oct 30 12:30:24.454102 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.454076    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5_odh-ai-gateway-infra(d59ed183-da6d-4132-a925-3a23af0aeb20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20"
Oct 30 12:30:24.456136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.456116    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"
Oct 30 12:30:24.456380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.456360    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl" event={"ID":"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1","Type":"ContainerDied","Data":"03b37a8b367b38e290b6929f10a4787ac4e25047bba1b2ca584cb65998b2c612"}
Oct 30 12:30:24.457946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.457883    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0ecae0-29889390-ts5gx"]
Oct 30 12:30:24.458350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.458331    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"
Oct 30 12:30:24.458625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.458604    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s" event={"ID":"d276acb6-2405-47d4-b385-0d293dc2ea38","Type":"ContainerDied","Data":"80a824ede2859c409d4048b89afb1c0e7499f1fbd5ce6805e4fd7c89372252e7"}
Oct 30 12:30:24.461810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.461261    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerStarted","Data":"ce42f36d7d92fbe354411b22854058c80bf9d6697a801563d6abba0764b56937"}
Oct 30 12:30:24.464572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.464242    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:24.467954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.466878    5060 generic.go:356] "Generic (PLEG): container finished" podID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerID="4dc204b9b79625e09931b0c3c6f77a50132084ca04298d907fecf4afe90c81bc" exitCode=60
Oct 30 12:30:24.467954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.466995    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" event={"ID":"5c9c1455-fce3-4161-a671-e3e509d5d854","Type":"ContainerDied","Data":"4dc204b9b79625e09931b0c3c6f77a50132084ca04298d907fecf4afe90c81bc"}
Oct 30 12:30:24.467954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.467478    5060 scope.go:122] "RemoveContainer" containerID="4dc204b9b79625e09931b0c3c6f77a50132084ca04298d907fecf4afe90c81bc"
Oct 30 12:30:24.467954 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.467905    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz_odh-ai-gateway-infra(5c9c1455-fce3-4161-a671-e3e509d5d854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854"
Oct 30 12:30:24.469996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.469103    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca\") pod \"d276acb6-2405-47d4-b385-0d293dc2ea38\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") "
Oct 30 12:30:24.469996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.469149    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8\" (UniqueName: \"kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8\") pod \"d276acb6-2405-47d4-b385-0d293dc2ea38\" (UID: \"d276acb6-2405-47d4-b385-0d293dc2ea38\") "
Oct 30 12:30:24.469996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.469282    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5\" (UniqueName: \"kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5\") pod \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") "
Oct 30 12:30:24.469996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.469401    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca\") pod \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\" (UID: \"3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1\") "
Oct 30 12:30:24.471753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.470961    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca" pod "3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" (UID: "3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:24.471753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.471289    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca" pod "d276acb6-2405-47d4-b385-0d293dc2ea38" (UID: "d276acb6-2405-47d4-b385-0d293dc2ea38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:24.475356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.474530    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5" pod "3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" (UID: "3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1"). InnerVolumeSpecName "kube-api-access-pkvn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:24.475356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.475126    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8" pod "d276acb6-2405-47d4-b385-0d293dc2ea38" (UID: "d276acb6-2405-47d4-b385-0d293dc2ea38"). InnerVolumeSpecName "kube-api-access-nl7r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:24.521541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.512180    5060 scope.go:122] "RemoveContainer" containerID="36fc1cffd92208b79b38263ad286c8d57523ce96104e90870a174cb588ee6488"
Oct 30 12:30:24.543446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.543387    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"]
Oct 30 12:30:24.545688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.544533    5060 generic.go:356] "Generic (PLEG): container finished" podID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerID="50927005db74351fefb143387f5ef7cfe13d86cef72519294ed7556a7fe7aebe" exitCode=60
Oct 30 12:30:24.545688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.544668    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerDied","Data":"50927005db74351fefb143387f5ef7cfe13d86cef72519294ed7556a7fe7aebe"}
Oct 30 12:30:24.545688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.545178    5060 scope.go:122] "RemoveContainer" containerID="50927005db74351fefb143387f5ef7cfe13d86cef72519294ed7556a7fe7aebe"
Oct 30 12:30:24.545688 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.545635    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92_odh-ai-gateway-infra(63ef289b-15ed-4151-be61-1b5727f8caf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5"
Oct 30 12:30:24.561034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.560898    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889390-vvxzr"]
Oct 30 12:30:24.581491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.576973    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276acb6-2405-47d4-b385-0d293dc2ea38-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:24.629591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.629555    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl7r8\" (UniqueName: \"kubernetes.io/projected/d276acb6-2405-47d4-b385-0d293dc2ea38-kube-api-access-nl7r8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:24.629591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.629592    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvn5\" (UniqueName: \"kubernetes.io/projected/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-kube-api-access-pkvn5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:24.629835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.629611    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:24.629835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.621517    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerID="049cc4e3b99a51053b6d27c77e2065d9857d093e0b3b9984baad205f498f5aee" exitCode=60
Oct 30 12:30:24.629835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.623338    5060 scope.go:122] "RemoveContainer" containerID="b6a6b8bf2427ec570a7cabf51c436b65ecc0d6d90a7fbdc2610e8651c5a2051a"
Oct 30 12:30:24.629835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.621548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerDied","Data":"049cc4e3b99a51053b6d27c77e2065d9857d093e0b3b9984baad205f498f5aee"}
Oct 30 12:30:24.631159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.630331    5060 scope.go:122] "RemoveContainer" containerID="049cc4e3b99a51053b6d27c77e2065d9857d093e0b3b9984baad205f498f5aee"
Oct 30 12:30:24.631159 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:24.630849    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j_odh-ai-gateway-infra(c4438f8b-b45c-4186-b63f-e9226d426ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8"
Oct 30 12:30:24.740809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.739260    5060 scope.go:122] "RemoveContainer" containerID="c0a7a9df3d596b9f0437544f14e084c4b279602ebbd38789acf736ab71ec2364"
Oct 30 12:30:24.909101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.907944    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"]
Oct 30 12:30:24.936223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.932953    5060 scope.go:122] "RemoveContainer" containerID="9f1a3dd280bb062d6a313b7005c9476ddd3ea65c28c1f16098aa75f23e22d2fb"
Oct 30 12:30:24.936223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.933345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:24.973494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.973029    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"]
Oct 30 12:30:24.999777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.993343    5060 patch_prober.go:29] interesting pod/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body=
Oct 30 12:30:24.999777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:24.993411    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused"
Oct 30 12:30:25.053754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.048405    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:25.057338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.057310    5060 scope.go:122] "RemoveContainer" containerID="dd0513fef3714d39f4945a63d1f01c7a0eda3e1c0982c7f42ec12bbfbb6caac2"
Oct 30 12:30:25.075040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.074928    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"]
Oct 30 12:30:25.098005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.096732    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca\") pod \"4d41c30a-79d8-463b-94a6-4da554dbf161\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") "
Oct 30 12:30:25.098005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.096975    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk\" (UniqueName: \"kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk\") pod \"4d41c30a-79d8-463b-94a6-4da554dbf161\" (UID: \"4d41c30a-79d8-463b-94a6-4da554dbf161\") "
Oct 30 12:30:25.122077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.119839    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca" pod "4d41c30a-79d8-463b-94a6-4da554dbf161" (UID: "4d41c30a-79d8-463b-94a6-4da554dbf161"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:25.122077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.121105    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk" pod "4d41c30a-79d8-463b-94a6-4da554dbf161" (UID: "4d41c30a-79d8-463b-94a6-4da554dbf161"). InnerVolumeSpecName "kube-api-access-94mqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:25.142102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.141870    5060 scope.go:122] "RemoveContainer" containerID="cb310af7271e4f32e440a53f0214a1ee321c27551fce31fbe4445efc7359eafc"
Oct 30 12:30:25.164167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.164081    5060 scope.go:122] "RemoveContainer" containerID="031446c081f23c7f2f4f97e5d6930b5a840e9b18f9385accfa45c980136eef23"
Oct 30 12:30:25.168280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.168256    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:25.222080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.213639    5060 scope.go:122] "RemoveContainer" containerID="0c7752bef0011a89cb5e6611f8210fb0a0b7b7e916f6f6bbceabd1aff1f5bc2b"
Oct 30 12:30:25.222080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.215661    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d41c30a-79d8-463b-94a6-4da554dbf161-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.222080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.215690    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mqk\" (UniqueName: \"kubernetes.io/projected/4d41c30a-79d8-463b-94a6-4da554dbf161-kube-api-access-94mqk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.277415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.272615    5060 scope.go:122] "RemoveContainer" containerID="e45bf1a4692bc3bc4ec90b0498c9b2721f4a01a5c5e1d636d35c336acf1b3ea4"
Oct 30 12:30:25.333762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.333728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh\" (UniqueName: \"kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh\") pod \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") "
Oct 30 12:30:25.335005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.334973    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca\") pod \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\" (UID: \"e3ba43b6-963c-4f0e-ab47-639eebd7497f\") "
Oct 30 12:30:25.338674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.338647    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca" pod "e3ba43b6-963c-4f0e-ab47-639eebd7497f" (UID: "e3ba43b6-963c-4f0e-ab47-639eebd7497f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:25.343113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.343069    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:25.344694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.344668    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh" pod "e3ba43b6-963c-4f0e-ab47-639eebd7497f" (UID: "e3ba43b6-963c-4f0e-ab47-639eebd7497f"). InnerVolumeSpecName "kube-api-access-hg5sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:25.437908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.437587    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca\") pod \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") "
Oct 30 12:30:25.437908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.437740    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52\" (UniqueName: \"kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52\") pod \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\" (UID: \"c89f2ac9-4e23-41f2-a8ae-596ce7000751\") "
Oct 30 12:30:25.440355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.440188    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg5sh\" (UniqueName: \"kubernetes.io/projected/e3ba43b6-963c-4f0e-ab47-639eebd7497f-kube-api-access-hg5sh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.440355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.440221    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ba43b6-963c-4f0e-ab47-639eebd7497f-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.441251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.441228    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca" pod "c89f2ac9-4e23-41f2-a8ae-596ce7000751" (UID: "c89f2ac9-4e23-41f2-a8ae-596ce7000751"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:25.559712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.510404    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr" podStartSLOduration=5.510380218 podStartE2EDuration="5.510380218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:25.394751418 +0000 UTC m=+3629.189635912" watchObservedRunningTime="2026-10-30 12:30:25.510380218 +0000 UTC m=+3629.305264712"
Oct 30 12:30:25.575555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.562567    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89f2ac9-4e23-41f2-a8ae-596ce7000751-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.593643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.590628    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52" pod "c89f2ac9-4e23-41f2-a8ae-596ce7000751" (UID: "c89f2ac9-4e23-41f2-a8ae-596ce7000751"). InnerVolumeSpecName "kube-api-access-rsh52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:25.612418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.612390    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:25.622548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.621821    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"]
Oct 30 12:30:25.635524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.635485    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"]
Oct 30 12:30:25.636692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.636670    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:25.639647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.639623    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:25.644339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.644312    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a1f322-29889390-jvd4s"]
Oct 30 12:30:25.664717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.664613    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"]
Oct 30 12:30:25.665092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.665059    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsh52\" (UniqueName: \"kubernetes.io/projected/c89f2ac9-4e23-41f2-a8ae-596ce7000751-kube-api-access-rsh52\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.684505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.682734    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc" event={"ID":"4d41c30a-79d8-463b-94a6-4da554dbf161","Type":"ContainerDied","Data":"1ca0380d15901d4921f7ed79327cf4662d847035053f8fcf126d3d662413334f"}
Oct 30 12:30:25.684505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.683241    5060 scope.go:122] "RemoveContainer" containerID="d8438507c54ec735e9b261ac5e3c32076b2090af5f16bd36e6b4f91505c13267"
Oct 30 12:30:25.684505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.683404    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"
Oct 30 12:30:25.696194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.696167    5060 generic.go:356] "Generic (PLEG): container finished" podID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerID="d2d35aef28dea0db50c977472cb78eae0fa12264f71830c54593c52397cb4f7f" exitCode=60
Oct 30 12:30:25.696411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.696259    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerDied","Data":"d2d35aef28dea0db50c977472cb78eae0fa12264f71830c54593c52397cb4f7f"}
Oct 30 12:30:25.696844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.696819    5060 scope.go:122] "RemoveContainer" containerID="d2d35aef28dea0db50c977472cb78eae0fa12264f71830c54593c52397cb4f7f"
Oct 30 12:30:25.697321 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:25.697292    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm_odh-ai-gateway-infra(7e5d41ef-26d9-4de3-886e-a1254ff2291f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f"
Oct 30 12:30:25.721952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.720317    5060 scope.go:122] "RemoveContainer" containerID="222068f2daa46fd328d01ec2f3cd89e0121a4bf7ccd963cd01c0553d7f55fd4f"
Oct 30 12:30:25.726080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.725566    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2" event={"ID":"e3ba43b6-963c-4f0e-ab47-639eebd7497f","Type":"ContainerDied","Data":"2d4b3c77b7743e40d0fd8104e9231e825479086808bf731ae4ae442135940d43"}
Oct 30 12:30:25.726080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.725693    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.768013    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s\" (UniqueName: \"kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s\") pod \"84f05789-dc11-4f6a-bccc-8d6c1740018e\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") "
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.768201    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca\") pod \"84f05789-dc11-4f6a-bccc-8d6c1740018e\" (UID: \"84f05789-dc11-4f6a-bccc-8d6c1740018e\") "
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.768293    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc\" (UniqueName: \"kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc\") pod \"292ec118-23ed-4030-843d-8a906ff083c7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") "
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.768485    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca\") pod \"292ec118-23ed-4030-843d-8a906ff083c7\" (UID: \"292ec118-23ed-4030-843d-8a906ff083c7\") "
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.768531    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca\") pod \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") "
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.768563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8\") pod \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\" (UID: \"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3\") "
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.781383    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8" pod "4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" (UID: "4a830ea6-1b84-4ddb-abe8-01830a9ce9e3"). InnerVolumeSpecName "kube-api-access-pnsm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.786802    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s" pod "84f05789-dc11-4f6a-bccc-8d6c1740018e" (UID: "84f05789-dc11-4f6a-bccc-8d6c1740018e"). InnerVolumeSpecName "kube-api-access-5qx7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:25.788331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.787160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca" pod "84f05789-dc11-4f6a-bccc-8d6c1740018e" (UID: "84f05789-dc11-4f6a-bccc-8d6c1740018e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:25.798970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.790002    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc" pod "292ec118-23ed-4030-843d-8a906ff083c7" (UID: "292ec118-23ed-4030-843d-8a906ff083c7"). InnerVolumeSpecName "kube-api-access-b74xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:25.798970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.790328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca" pod "292ec118-23ed-4030-843d-8a906ff083c7" (UID: "292ec118-23ed-4030-843d-8a906ff083c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:25.798970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.790669    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca" pod "4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" (UID: "4a830ea6-1b84-4ddb-abe8-01830a9ce9e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:25.810898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.810791    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"]
Oct 30 12:30:25.856717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.856610    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"]
Oct 30 12:30:25.884481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.881327    5060 scope.go:122] "RemoveContainer" containerID="335df6cf1c7048690b6da945c12f4751eab7780a84d1e577eb0a406c5a30e8b6"
Oct 30 12:30:25.893145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.892335    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qx7s\" (UniqueName: \"kubernetes.io/projected/84f05789-dc11-4f6a-bccc-8d6c1740018e-kube-api-access-5qx7s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.893145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.892363    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f05789-dc11-4f6a-bccc-8d6c1740018e-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.893145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.892379    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74xc\" (UniqueName: \"kubernetes.io/projected/292ec118-23ed-4030-843d-8a906ff083c7-kube-api-access-b74xc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.893145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.892394    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/292ec118-23ed-4030-843d-8a906ff083c7-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.893145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.892408    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.893145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.892423    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3-kube-api-access-pnsm8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:25.904215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.900331    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528" event={"ID":"84f05789-dc11-4f6a-bccc-8d6c1740018e","Type":"ContainerDied","Data":"02a639c1726b8e9c88db111321d0965d4415b894b7ec7499f6dca3d3db46213d"}
Oct 30 12:30:25.904215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.900484    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"
Oct 30 12:30:25.914052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.913699    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6c1d179-d831-4a74-a377-1d549e622017" containerID="ce42f36d7d92fbe354411b22854058c80bf9d6697a801563d6abba0764b56937" exitCode=60
Oct 30 12:30:25.914052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.913815    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerDied","Data":"ce42f36d7d92fbe354411b22854058c80bf9d6697a801563d6abba0764b56937"}
Oct 30 12:30:25.919477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.917390    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"
Oct 30 12:30:25.927204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.926962    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7" event={"ID":"292ec118-23ed-4030-843d-8a906ff083c7","Type":"ContainerDied","Data":"a593dcf6679734dade94d5c1568ed8bd264092037df794a6720f7b332f6ff37e"}
Oct 30 12:30:25.929651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.929217    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2" event={"ID":"c89f2ac9-4e23-41f2-a8ae-596ce7000751","Type":"ContainerDied","Data":"4b3cb69ca096fd5156b4acba21b95e318638a7ba4e2542a17fd351764ea4444e"}
Oct 30 12:30:25.929651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.929308    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"
Oct 30 12:30:25.941855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.941019    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp" event={"ID":"4a830ea6-1b84-4ddb-abe8-01830a9ce9e3","Type":"ContainerDied","Data":"e2a5e6b0aa24e448a246cb66bcb0faec69c036d366cd9e763285113e34911a0a"}
Oct 30 12:30:25.941855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.941257    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"
Oct 30 12:30:25.963220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.962728    5060 scope.go:122] "RemoveContainer" containerID="11fc25f7a9971b0f45e4d4af1bf4d1ce633579e93fdcf400278178b0fb84a271"
Oct 30 12:30:25.972399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.972348    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"]
Oct 30 12:30:25.991826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:25.991741    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-933195-29889390-7n8tl"]
Oct 30 12:30:26.001181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.001143    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"]
Oct 30 12:30:26.015147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.008156    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:30:26.015147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.010735    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:30:26.015147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.014198    5060 scope.go:122] "RemoveContainer" containerID="f7b947dfb907bd4a4e2de136002f5f5b10feb418ff5ef2128f50568d32635b96"
Oct 30 12:30:26.024998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.024542    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:30:26.078166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.077055    5060 scope.go:122] "RemoveContainer" containerID="6d760ad873e846b418f3ee1d55a0df513100d5f1f2ca3bdbd675e38e59767f94"
Oct 30 12:30:26.200693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.199132    5060 scope.go:122] "RemoveContainer" containerID="d27b4b9dec0b256c8e5f9411c09569c1443ce0e87f56656deaa270ad8bf32baa"
Oct 30 12:30:26.205634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.205337    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:26.246338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.245809    5060 scope.go:122] "RemoveContainer" containerID="821d7c57fa0648c0bcac593586bb1b17ae9fc2a743093c5f9fc11ec4d32f9fdc"
Oct 30 12:30:26.248200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.248120    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:26.307394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.306511    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2\" (UniqueName: \"kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2\") pod \"4e468f83-2204-4f1d-9a08-f41115e4ef86\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") "
Oct 30 12:30:26.307394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.306616    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca\") pod \"4e468f83-2204-4f1d-9a08-f41115e4ef86\" (UID: \"4e468f83-2204-4f1d-9a08-f41115e4ef86\") "
Oct 30 12:30:26.307394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.307347    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca" pod "4e468f83-2204-4f1d-9a08-f41115e4ef86" (UID: "4e468f83-2204-4f1d-9a08-f41115e4ef86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.345623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.334504    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2" pod "4e468f83-2204-4f1d-9a08-f41115e4ef86" (UID: "4e468f83-2204-4f1d-9a08-f41115e4ef86"). InnerVolumeSpecName "kube-api-access-vlrd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.408981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.408544    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc\" (UniqueName: \"kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc\") pod \"d59ed183-da6d-4132-a925-3a23af0aeb20\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") "
Oct 30 12:30:26.409527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.409500    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca\") pod \"d59ed183-da6d-4132-a925-3a23af0aeb20\" (UID: \"d59ed183-da6d-4132-a925-3a23af0aeb20\") "
Oct 30 12:30:26.410614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.410593    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrd2\" (UniqueName: \"kubernetes.io/projected/4e468f83-2204-4f1d-9a08-f41115e4ef86-kube-api-access-vlrd2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.410734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.410721    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468f83-2204-4f1d-9a08-f41115e4ef86-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.411142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.411123    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca" pod "d59ed183-da6d-4132-a925-3a23af0aeb20" (UID: "d59ed183-da6d-4132-a925-3a23af0aeb20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.414363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.414339    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc" pod "d59ed183-da6d-4132-a925-3a23af0aeb20" (UID: "d59ed183-da6d-4132-a925-3a23af0aeb20"). InnerVolumeSpecName "kube-api-access-s89pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.432543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.432198    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" path="/var/lib/kubelet/pods/236a39f6-e955-4e84-b437-7592c13f8f76/volumes"
Oct 30 12:30:26.434215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.433134    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" path="/var/lib/kubelet/pods/3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1/volumes"
Oct 30 12:30:26.434215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.433893    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" path="/var/lib/kubelet/pods/d276acb6-2405-47d4-b385-0d293dc2ea38/volumes"
Oct 30 12:30:26.435346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.435066    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" path="/var/lib/kubelet/pods/e03a2654-376e-4fb8-a9fd-2b5bf73e3f11/volumes"
Oct 30 12:30:26.436195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.435836    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" path="/var/lib/kubelet/pods/f0a6acb4-68df-4d60-add2-101fe1ddf61e/volumes"
Oct 30 12:30:26.477097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.467273    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:26.497829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.497800    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"]
Oct 30 12:30:26.510114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.510084    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:26.518249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.518218    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889390-gl528"]
Oct 30 12:30:26.521308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.521275    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59ed183-da6d-4132-a925-3a23af0aeb20-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.521308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.521305    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s89pc\" (UniqueName: \"kubernetes.io/projected/d59ed183-da6d-4132-a925-3a23af0aeb20-kube-api-access-s89pc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.522255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.522031    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:26.530702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.530678    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:26.541378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.540701    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:26.558258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.558187    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"]
Oct 30 12:30:26.564867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.564815    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bbae89-29889390-scmgc"]
Oct 30 12:30:26.584921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.584877    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"]
Oct 30 12:30:26.590873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.590841    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889390-w2rk2"]
Oct 30 12:30:26.622543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.622503    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca\") pod \"a408b674-92df-4821-b568-7f6ee75c5d36\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") "
Oct 30 12:30:26.622857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.622834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca\") pod \"a6c1d179-d831-4a74-a377-1d549e622017\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") "
Oct 30 12:30:26.623074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.623044    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh\" (UniqueName: \"kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh\") pod \"0a89196e-f4d8-436e-8bce-144bd3164a03\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") "
Oct 30 12:30:26.628754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.628718    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8\" (UniqueName: \"kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8\") pod \"a6c1d179-d831-4a74-a377-1d549e622017\" (UID: \"a6c1d179-d831-4a74-a377-1d549e622017\") "
Oct 30 12:30:26.628925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.628822    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj\" (UniqueName: \"kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj\") pod \"a408b674-92df-4821-b568-7f6ee75c5d36\" (UID: \"a408b674-92df-4821-b568-7f6ee75c5d36\") "
Oct 30 12:30:26.629014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.628930    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw\" (UniqueName: \"kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw\") pod \"5c9c1455-fce3-4161-a671-e3e509d5d854\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") "
Oct 30 12:30:26.629014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.628995    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca\") pod \"0a89196e-f4d8-436e-8bce-144bd3164a03\" (UID: \"0a89196e-f4d8-436e-8bce-144bd3164a03\") "
Oct 30 12:30:26.629140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.629072    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca\") pod \"63ef289b-15ed-4151-be61-1b5727f8caf5\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") "
Oct 30 12:30:26.629140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.629105    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca\") pod \"5c9c1455-fce3-4161-a671-e3e509d5d854\" (UID: \"5c9c1455-fce3-4161-a671-e3e509d5d854\") "
Oct 30 12:30:26.629335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.629300    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq\" (UniqueName: \"kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq\") pod \"63ef289b-15ed-4151-be61-1b5727f8caf5\" (UID: \"63ef289b-15ed-4151-be61-1b5727f8caf5\") "
Oct 30 12:30:26.630882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.623526    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca" pod "a408b674-92df-4821-b568-7f6ee75c5d36" (UID: "a408b674-92df-4821-b568-7f6ee75c5d36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.630882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.623588    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca" pod "a6c1d179-d831-4a74-a377-1d549e622017" (UID: "a6c1d179-d831-4a74-a377-1d549e622017"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.631863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.631644    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca" pod "5c9c1455-fce3-4161-a671-e3e509d5d854" (UID: "5c9c1455-fce3-4161-a671-e3e509d5d854"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.632965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.632101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca" pod "0a89196e-f4d8-436e-8bce-144bd3164a03" (UID: "0a89196e-f4d8-436e-8bce-144bd3164a03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.633256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.633193    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8" pod "a6c1d179-d831-4a74-a377-1d549e622017" (UID: "a6c1d179-d831-4a74-a377-1d549e622017"). InnerVolumeSpecName "kube-api-access-mrqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.633256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.633222    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca" pod "63ef289b-15ed-4151-be61-1b5727f8caf5" (UID: "63ef289b-15ed-4151-be61-1b5727f8caf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:26.634576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.634542    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj" pod "a408b674-92df-4821-b568-7f6ee75c5d36" (UID: "a408b674-92df-4821-b568-7f6ee75c5d36"). InnerVolumeSpecName "kube-api-access-j58rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.634803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.634783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh" pod "0a89196e-f4d8-436e-8bce-144bd3164a03" (UID: "0a89196e-f4d8-436e-8bce-144bd3164a03"). InnerVolumeSpecName "kube-api-access-hhsfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.635083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.635059    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw" pod "5c9c1455-fce3-4161-a671-e3e509d5d854" (UID: "5c9c1455-fce3-4161-a671-e3e509d5d854"). InnerVolumeSpecName "kube-api-access-dn7xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.635861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.635841    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq" pod "63ef289b-15ed-4151-be61-1b5727f8caf5" (UID: "63ef289b-15ed-4151-be61-1b5727f8caf5"). InnerVolumeSpecName "kube-api-access-ftngq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:26.652901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.652865    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"]
Oct 30 12:30:26.661007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.660906    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-67e819-29889390-76nw7"]
Oct 30 12:30:26.668241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.668211    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735136    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a89196e-f4d8-436e-8bce-144bd3164a03-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735177    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef289b-15ed-4151-be61-1b5727f8caf5-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735193    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1455-fce3-4161-a671-e3e509d5d854-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735207    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftngq\" (UniqueName: \"kubernetes.io/projected/63ef289b-15ed-4151-be61-1b5727f8caf5-kube-api-access-ftngq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735223    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a408b674-92df-4821-b568-7f6ee75c5d36-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735237    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c1d179-d831-4a74-a377-1d549e622017-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735250    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhsfh\" (UniqueName: \"kubernetes.io/projected/0a89196e-f4d8-436e-8bce-144bd3164a03-kube-api-access-hhsfh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735263    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrqt8\" (UniqueName: \"kubernetes.io/projected/a6c1d179-d831-4a74-a377-1d549e622017-kube-api-access-mrqt8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735278    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58rj\" (UniqueName: \"kubernetes.io/projected/a408b674-92df-4821-b568-7f6ee75c5d36-kube-api-access-j58rj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.735324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.735293    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn7xw\" (UniqueName: \"kubernetes.io/projected/5c9c1455-fce3-4161-a671-e3e509d5d854-kube-api-access-dn7xw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:26.799425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.799360    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"]
Oct 30 12:30:26.841436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.841321    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889390-kjfzp"]
Oct 30 12:30:26.856630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.856600    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"]
Oct 30 12:30:26.869157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.869125    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-849669-29889390-qnfk2"]
Oct 30 12:30:26.960646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.960615    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"
Oct 30 12:30:26.960866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.960637    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92" event={"ID":"63ef289b-15ed-4151-be61-1b5727f8caf5","Type":"ContainerDied","Data":"42094b881802314fecc3ddb41c6d00f701bae16c5837aa08f4f7ffa2fe9ea02e"}
Oct 30 12:30:26.960866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.960700    5060 scope.go:122] "RemoveContainer" containerID="50927005db74351fefb143387f5ef7cfe13d86cef72519294ed7556a7fe7aebe"
Oct 30 12:30:26.969561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.968107    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"
Oct 30 12:30:26.969561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.968120    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr" event={"ID":"0a89196e-f4d8-436e-8bce-144bd3164a03","Type":"ContainerDied","Data":"b828d130a2b9556a229b376a0ba51a4571a23c1afb628c39f3e4da53f7d994d4"}
Oct 30 12:30:26.971326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.971298    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z" event={"ID":"a408b674-92df-4821-b568-7f6ee75c5d36","Type":"ContainerDied","Data":"06d884301d6de4de91e661f63dec8ee0eccf257e7ecf67c271570d007f66cd36"}
Oct 30 12:30:26.971496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.971375    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"
Oct 30 12:30:26.974150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.973561    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"
Oct 30 12:30:26.974150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.973612    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8" event={"ID":"4e468f83-2204-4f1d-9a08-f41115e4ef86","Type":"ContainerDied","Data":"e35b930a00fcbb02f4803483688311767f27d90edcd94416857c983a7d2e0435"}
Oct 30 12:30:26.977892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.977839    5060 scope.go:122] "RemoveContainer" containerID="85322989ca7ea21d4b02b82b7eddd24001912ba6aa3d2e8bd83c39e74ad86d67"
Oct 30 12:30:26.983548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.983014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5" event={"ID":"d59ed183-da6d-4132-a925-3a23af0aeb20","Type":"ContainerDied","Data":"d9650039aca80258c60309b5e485ddbf93b6d5c9110f3cf9c68501f021c314b8"}
Oct 30 12:30:26.983548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.983143    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"
Oct 30 12:30:26.988847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.988823    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9" event={"ID":"a6c1d179-d831-4a74-a377-1d549e622017","Type":"ContainerDied","Data":"505e49dd67a2351fe3039371c78026a0bfe1f9ade7f3857f31cba9f61274ee2b"}
Oct 30 12:30:26.989064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.989046    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"
Oct 30 12:30:27.002795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:26.999372    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"
Oct 30 12:30:27.006142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.004540    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz" event={"ID":"5c9c1455-fce3-4161-a671-e3e509d5d854","Type":"ContainerDied","Data":"69b5ff5fa51a9fabcf0cc086b9959c6e237a421d51d7d23cbed077f2f52c9b3b"}
Oct 30 12:30:27.010370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.010258    5060 scope.go:122] "RemoveContainer" containerID="8ea9cd69b9b5823abcbb855628a9cb7789396499a1c2caa894db804580e625db"
Oct 30 12:30:27.028442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.028189    5060 scope.go:122] "RemoveContainer" containerID="bb82036947457e663de68a362f62561dc269fcea8c9b21bd78afc89be8cd6ceb"
Oct 30 12:30:27.041685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.041660    5060 scope.go:122] "RemoveContainer" containerID="f0887ecd2900b945448d4680b46dd0acd1149f02718d52295d91808e45e87403"
Oct 30 12:30:27.056852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.056823    5060 scope.go:122] "RemoveContainer" containerID="ce42f36d7d92fbe354411b22854058c80bf9d6697a801563d6abba0764b56937"
Oct 30 12:30:27.077698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.077252    5060 scope.go:122] "RemoveContainer" containerID="4dc204b9b79625e09931b0c3c6f77a50132084ca04298d907fecf4afe90c81bc"
Oct 30 12:30:27.161520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.158670    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"]
Oct 30 12:30:27.274247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.274127    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:27.426172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.426116    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"]
Oct 30 12:30:27.431591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.431558    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b8a9d-29889390-5tv92"]
Oct 30 12:30:27.459104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.459078    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca\") pod \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") "
Oct 30 12:30:27.459519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.459504    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb\" (UniqueName: \"kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb\") pod \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\" (UID: \"7e5d41ef-26d9-4de3-886e-a1254ff2291f\") "
Oct 30 12:30:27.463539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.463163    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca" pod "7e5d41ef-26d9-4de3-886e-a1254ff2291f" (UID: "7e5d41ef-26d9-4de3-886e-a1254ff2291f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:27.465973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.465927    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb" pod "7e5d41ef-26d9-4de3-886e-a1254ff2291f" (UID: "7e5d41ef-26d9-4de3-886e-a1254ff2291f"). InnerVolumeSpecName "kube-api-access-5j2wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:27.468559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.468534    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:27.509679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.509589    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"]
Oct 30 12:30:27.521509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.520947    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889390-5zhdz"]
Oct 30 12:30:27.554314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.554280    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"]
Oct 30 12:30:27.560929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.560889    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889390-spnj5"]
Oct 30 12:30:27.562143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.562115    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j2wb\" (UniqueName: \"kubernetes.io/projected/7e5d41ef-26d9-4de3-886e-a1254ff2291f-kube-api-access-5j2wb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:27.562267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.562148    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5d41ef-26d9-4de3-886e-a1254ff2291f-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:27.594287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.594249    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"]
Oct 30 12:30:27.599366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.599334    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0190d8-29889390-2t4g8"]
Oct 30 12:30:27.633090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.633047    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"]
Oct 30 12:30:27.638539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.638508    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-737f64-29889390-rjhw9"]
Oct 30 12:30:27.663404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.663371    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9\" (UniqueName: \"kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9\") pod \"c4438f8b-b45c-4186-b63f-e9226d426ce8\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") "
Oct 30 12:30:27.663613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.663585    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca\") pod \"c4438f8b-b45c-4186-b63f-e9226d426ce8\" (UID: \"c4438f8b-b45c-4186-b63f-e9226d426ce8\") "
Oct 30 12:30:27.663928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.663903    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca" pod "c4438f8b-b45c-4186-b63f-e9226d426ce8" (UID: "c4438f8b-b45c-4186-b63f-e9226d426ce8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:27.664268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.664251    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4438f8b-b45c-4186-b63f-e9226d426ce8-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:27.665949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.665925    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9" pod "c4438f8b-b45c-4186-b63f-e9226d426ce8" (UID: "c4438f8b-b45c-4186-b63f-e9226d426ce8"). InnerVolumeSpecName "kube-api-access-9rkn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:27.719862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.719792    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"]
Oct 30 12:30:27.726438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.726390    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889390-hlx7z"]
Oct 30 12:30:27.755025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.754988    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"]
Oct 30 12:30:27.760108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.760037    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae3538-29889390-z5klr"]
Oct 30 12:30:27.766500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:27.766445    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rkn9\" (UniqueName: \"kubernetes.io/projected/c4438f8b-b45c-4186-b63f-e9226d426ce8-kube-api-access-9rkn9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:28.021914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.021821    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm" event={"ID":"7e5d41ef-26d9-4de3-886e-a1254ff2291f","Type":"ContainerDied","Data":"7b7087004cc48af5291c4c6641a65d914be531a4e8787fe57d2befc704b23407"}
Oct 30 12:30:28.021914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.021848    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"
Oct 30 12:30:28.022173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.021928    5060 scope.go:122] "RemoveContainer" containerID="d2d35aef28dea0db50c977472cb78eae0fa12264f71830c54593c52397cb4f7f"
Oct 30 12:30:28.030014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.029988    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"
Oct 30 12:30:28.030159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.029989    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j" event={"ID":"c4438f8b-b45c-4186-b63f-e9226d426ce8","Type":"ContainerDied","Data":"e2e46e0718dd230a769e0829ec79831594c1d2cda1ac311f9aebf6bb24d23912"}
Oct 30 12:30:28.039283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.039261    5060 scope.go:122] "RemoveContainer" containerID="049cc4e3b99a51053b6d27c77e2065d9857d093e0b3b9984baad205f498f5aee"
Oct 30 12:30:28.099593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.099551    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"]
Oct 30 12:30:28.105145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.105110    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889390-lttdm"]
Oct 30 12:30:28.133779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.133740    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"]
Oct 30 12:30:28.138942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.138905    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-28ca05-29889390-xdz8j"]
Oct 30 12:30:28.337380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.337280    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" path="/var/lib/kubelet/pods/0a89196e-f4d8-436e-8bce-144bd3164a03/volumes"
Oct 30 12:30:28.337853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.337823    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292ec118-23ed-4030-843d-8a906ff083c7" path="/var/lib/kubelet/pods/292ec118-23ed-4030-843d-8a906ff083c7/volumes"
Oct 30 12:30:28.338235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.338221    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" path="/var/lib/kubelet/pods/4a830ea6-1b84-4ddb-abe8-01830a9ce9e3/volumes"
Oct 30 12:30:28.338881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.338859    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" path="/var/lib/kubelet/pods/4d41c30a-79d8-463b-94a6-4da554dbf161/volumes"
Oct 30 12:30:28.339271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.339256    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" path="/var/lib/kubelet/pods/4e468f83-2204-4f1d-9a08-f41115e4ef86/volumes"
Oct 30 12:30:28.339710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.339692    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" path="/var/lib/kubelet/pods/5c9c1455-fce3-4161-a671-e3e509d5d854/volumes"
Oct 30 12:30:28.340378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.340359    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" path="/var/lib/kubelet/pods/63ef289b-15ed-4151-be61-1b5727f8caf5/volumes"
Oct 30 12:30:28.340827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.340808    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" path="/var/lib/kubelet/pods/7e5d41ef-26d9-4de3-886e-a1254ff2291f/volumes"
Oct 30 12:30:28.341196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.341180    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" path="/var/lib/kubelet/pods/84f05789-dc11-4f6a-bccc-8d6c1740018e/volumes"
Oct 30 12:30:28.341904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.341884    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" path="/var/lib/kubelet/pods/a408b674-92df-4821-b568-7f6ee75c5d36/volumes"
Oct 30 12:30:28.342271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.342255    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c1d179-d831-4a74-a377-1d549e622017" path="/var/lib/kubelet/pods/a6c1d179-d831-4a74-a377-1d549e622017/volumes"
Oct 30 12:30:28.342736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.342715    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" path="/var/lib/kubelet/pods/c4438f8b-b45c-4186-b63f-e9226d426ce8/volumes"
Oct 30 12:30:28.343723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.343699    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" path="/var/lib/kubelet/pods/c89f2ac9-4e23-41f2-a8ae-596ce7000751/volumes"
Oct 30 12:30:28.344289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.344268    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" path="/var/lib/kubelet/pods/d59ed183-da6d-4132-a925-3a23af0aeb20/volumes"
Oct 30 12:30:28.345299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.345276    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" path="/var/lib/kubelet/pods/e3ba43b6-963c-4f0e-ab47-639eebd7497f/volumes"
Oct 30 12:30:28.456799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:28.456761    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:30.054399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.054361    5060 generic.go:356] "Generic (PLEG): container finished" podID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerID="b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085" exitCode=0
Oct 30 12:30:30.054875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.054432    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" event={"ID":"68bf4d13-9763-4407-9cd5-0fa4bbb628b1","Type":"ContainerDied","Data":"b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"}
Oct 30 12:30:30.054969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.054953    5060 scope.go:122] "RemoveContainer" containerID="b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"
Oct 30 12:30:30.057174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.057147    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:30:30.057262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.057213    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:30:30.057262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.057248    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:30:30.115264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:30.115229    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:31.068001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.067967    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" event={"ID":"68bf4d13-9763-4407-9cd5-0fa4bbb628b1","Type":"ContainerStarted","Data":"863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c"}
Oct 30 12:30:31.075600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.075570    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:30:31.080426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.080388    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 12:30:31.080687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.080472    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 12:30:31.464638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.464595    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:31.496481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.488829    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:31.972084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:31.971867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:32.075496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:32.075440    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 12:30:32.075906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:32.075538    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 12:30:32.130122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:32.130090    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:30:33.032009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:33.031973    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"]
Oct 30 12:30:33.044142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:33.044111    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f9d52c0-dst5q"]
Oct 30 12:30:33.075535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:33.075505    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 12:30:33.075942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:33.075559    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 12:30:33.629828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:33.629740    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:33.662144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:33.662104    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:30:34.075460 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:34.075419    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 12:30:34.075654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:34.075525    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 12:30:34.337711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:34.337632    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e328790-11da-4d08-b07b-66c9ab6b522b" path="/var/lib/kubelet/pods/8e328790-11da-4d08-b07b-66c9ab6b522b/volumes"
Oct 30 12:30:35.075678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:35.075643    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 12:30:35.076085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:35.075712    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 12:30:35.360877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:35.360794    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:36.031598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:36.031559    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"]
Oct 30 12:30:36.040280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:36.040246    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a565d71d-tp9dp"]
Oct 30 12:30:36.076475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:36.076418    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 12:30:36.077028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:36.076537    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 12:30:36.354864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:36.354755    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2042a26-1cce-4243-af82-77c466a5a6bf" path="/var/lib/kubelet/pods/f2042a26-1cce-4243-af82-77c466a5a6bf/volumes"
Oct 30 12:30:36.932420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:36.932378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.080703    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"]
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081885    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081908    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081923    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081931    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081942    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081950    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081964    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081972    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081984    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.081991    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082005    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082012    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082024    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082031    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082042    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082048    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082060    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082067    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082080    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082087    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082098    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082106    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082115    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082122    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082139    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082146    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082155    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082162    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082170    5060 cpu_manager.go:414] "containerMap: removing container" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082178    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082195    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082204    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082216    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082223    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082248    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082257    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082268    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082275    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082290    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082298    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082311    5060 cpu_manager.go:414] "containerMap: removing container" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082317    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082327    5060 cpu_manager.go:414] "containerMap: removing container" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082334    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082343    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082350    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082360    5060 cpu_manager.go:414] "containerMap: removing container" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082368    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082380    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082387    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082397    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082405    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082416    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082423    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082434    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082440    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082471    5060 cpu_manager.go:414] "containerMap: removing container" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082479    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082492    5060 cpu_manager.go:414] "containerMap: removing container" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082501    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082516    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082523    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082533    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082540    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082554    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082561    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082579    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082586    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082603    5060 cpu_manager.go:414] "containerMap: removing container" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082610    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082624    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082630    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082642    5060 cpu_manager.go:414] "containerMap: removing container" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082649    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082662    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082668    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082680    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082687    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082695    5060 cpu_manager.go:414] "containerMap: removing container" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082701    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082710    5060 cpu_manager.go:414] "containerMap: removing container" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082717    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082729    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082737    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082751    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082757    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082767    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082775    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082798    5060 cpu_manager.go:414] "containerMap: removing container" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082805    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082817    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082824    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082838    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082845    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082857    5060 cpu_manager.go:414] "containerMap: removing container" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082864    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082873    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082881    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082890    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082896    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082913    5060 cpu_manager.go:414] "containerMap: removing container" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082920    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082930    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082936    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082953    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082960    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082973    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082979    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.082994    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083001    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083010    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083016    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083216    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083236    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083244    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083253    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083262    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083271    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083280    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083290    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083300    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083308    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083315    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083327    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083334    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083341    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083351    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083358    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083366    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083373    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083381    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083392    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083401    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083409    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083418    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083429    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083439    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083446    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083474    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083485    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083494    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083504    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083533    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083545    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083553    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083562    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083570    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083578    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083586    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083595    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083603    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083611    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083619    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083627    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:37.083507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083637    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="236a39f6-e955-4e84-b437-7592c13f8f76" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083647    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083657    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083667    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083675    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083684    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083693    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083702    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083710    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083720    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083733    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083740    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083749    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ef289b-15ed-4151-be61-1b5727f8caf5" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.083759    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.128945    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.129305    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.145683    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd0ff7\"/\"kube-root-ca.crt\""
Oct 30 12:30:37.148844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.148102    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"]
Oct 30 12:30:37.186537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.186267    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd0ff7\"/\"default-dockercfg-whccl\""
Oct 30 12:30:37.186537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.186383    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd0ff7\"/\"test-model-dd0ff7-kserve-self-signed-certs\""
Oct 30 12:30:37.186537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.186493    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd0ff7\"/\"openshift-service-ca.crt\""
Oct 30 12:30:37.205865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.205755    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:30:37.403726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.403613    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:37.462491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.460304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.462491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.460354    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.462491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.460408    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.462491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.460492    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnsp\" (UniqueName: \"kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.462491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.460572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.462491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.460849    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.585416    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnsp\" (UniqueName: \"kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.585528    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.585730    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.585904    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.585930    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.585966    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.590585    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.590815    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.593485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.591083    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.598742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.598717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.614812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.614691    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnsp\" (UniqueName: \"kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.616116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.616049    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs\") pod \"test-model-dd0ff7-kserve-6f5d564d69-qdh2j\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") " pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.718964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.713277    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:37.765099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.764739    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:37.958379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:37.872955    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:38.024475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.023178    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:38.083478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.074706    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"]
Oct 30 12:30:38.167479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.166013    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f322-49wvh"]
Oct 30 12:30:38.343579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.343023    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c98574-8d1f-4844-a8c1-62cdda93fbe0" path="/var/lib/kubelet/pods/c2c98574-8d1f-4844-a8c1-62cdda93fbe0/volumes"
Oct 30 12:30:38.500394 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:38.500360    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d00dad_63da_4d02_9fd1_a80b159a44cb.slice/crio-873d0454856db02a15fc74872b616d4e5a57ac9a4506a138ed6f4f1bdb9fa0ea WatchSource:0}: Error finding container 873d0454856db02a15fc74872b616d4e5a57ac9a4506a138ed6f4f1bdb9fa0ea: Status 404 returned error can't find the container with id 873d0454856db02a15fc74872b616d4e5a57ac9a4506a138ed6f4f1bdb9fa0ea
Oct 30 12:30:38.505374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.505346    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"]
Oct 30 12:30:38.526128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.526007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:38.594791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.593739    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"]
Oct 30 12:30:38.675478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:38.675401    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:39.173347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.173309    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"]
Oct 30 12:30:39.174089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174073    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:39.174132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174091    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a830ea6-1b84-4ddb-abe8-01830a9ce9e3" containerName="cleanup"
Oct 30 12:30:39.174132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174107    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:39.174132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174112    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d41c30a-79d8-463b-94a6-4da554dbf161" containerName="cleanup"
Oct 30 12:30:39.174132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174131    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174137    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89f2ac9-4e23-41f2-a8ae-596ce7000751" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174156    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174160    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e468f83-2204-4f1d-9a08-f41115e4ef86" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174173    5060 cpu_manager.go:414] "containerMap: removing container" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174177    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174190    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174195    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaeb2e3-8d77-491b-a5e1-e9df6e24ede1" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174205    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174209    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174217    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174221    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174232    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174236    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03a2654-376e-4fb8-a9fd-2b5bf73e3f11" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174245    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174248    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174255    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174259    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a408b674-92df-4821-b568-7f6ee75c5d36" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174278    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:39.174279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174283    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174290    5060 cpu_manager.go:414] "containerMap: removing container" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174295    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144e4de0-8203-4e08-a384-f6a89b6627a5" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174400    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4438f8b-b45c-4186-b63f-e9226d426ce8" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174410    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="292ec118-23ed-4030-843d-8a906ff083c7" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174420    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f05789-dc11-4f6a-bccc-8d6c1740018e" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174426    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59ed183-da6d-4132-a925-3a23af0aeb20" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174431    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5d41ef-26d9-4de3-886e-a1254ff2291f" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174437    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a6acb4-68df-4d60-add2-101fe1ddf61e" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174445    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd8aeb3-aeb0-49de-93b9-29358f4e5ba8" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174465    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9c1455-fce3-4161-a671-e3e509d5d854" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174472    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d276acb6-2405-47d4-b385-0d293dc2ea38" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174479    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ba43b6-963c-4f0e-ab47-639eebd7497f" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174486    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a89196e-f4d8-436e-8bce-144bd3164a03" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174492    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c1d179-d831-4a74-a377-1d549e622017" containerName="cleanup"
Oct 30 12:30:39.174924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.174497    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed245e9-ef5a-4a2d-afda-2b1fc7bf9096" containerName="cleanup"
Oct 30 12:30:39.178515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.178489    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.195709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.195681    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"]
Oct 30 12:30:39.201954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.201316    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:30:39.201954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.201474    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:30:39.201954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.201579    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:30:39.214492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.211996    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:30:39.281510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.281327    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.281736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.281550    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.281736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.281700    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmf4\" (UniqueName: \"kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.382991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.382943    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.383203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.383095    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.383283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.383259    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmf4\" (UniqueName: \"kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.383910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.383882    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.386098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.386074    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.407733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.407700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmf4\" (UniqueName: \"kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4\") pod \"maas-api-revoke-sub-ca070d4d-2rs87\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.429506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.429348    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j" event={"ID":"c0d00dad-63da-4d02-9fd1-a80b159a44cb","Type":"ContainerStarted","Data":"11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512"}
Oct 30 12:30:39.429506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.429405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j" event={"ID":"c0d00dad-63da-4d02-9fd1-a80b159a44cb","Type":"ContainerStarted","Data":"873d0454856db02a15fc74872b616d4e5a57ac9a4506a138ed6f4f1bdb9fa0ea"}
Oct 30 12:30:39.476185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.476061    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j" podStartSLOduration=2.476040625 podStartE2EDuration="2.476040625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:39.471119298 +0000 UTC m=+3643.266003796" watchObservedRunningTime="2026-10-30 12:30:39.476040625 +0000 UTC m=+3643.270925117"
Oct 30 12:30:39.500525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.500491    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:39.765925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.765894    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:39.823928 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:39.823893    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b748546_4ce6_461f_ae9a_5788fd038e6e.slice/crio-df5b98add957b8a7361b035f04fee1a386c8a05da94498a722d5b1f7cb574374 WatchSource:0}: Error finding container df5b98add957b8a7361b035f04fee1a386c8a05da94498a722d5b1f7cb574374: Status 404 returned error can't find the container with id df5b98add957b8a7361b035f04fee1a386c8a05da94498a722d5b1f7cb574374
Oct 30 12:30:39.836138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.828105    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"]
Oct 30 12:30:39.977232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:39.977190    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:40.442515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:40.442467    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87" event={"ID":"6b748546-4ce6-461f-ae9a-5788fd038e6e","Type":"ContainerStarted","Data":"176125918f676537bb681b30756aa60cf992b79f02ef618d11194e85ce6662e4"}
Oct 30 12:30:40.443106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:40.442544    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87" event={"ID":"6b748546-4ce6-461f-ae9a-5788fd038e6e","Type":"ContainerStarted","Data":"df5b98add957b8a7361b035f04fee1a386c8a05da94498a722d5b1f7cb574374"}
Oct 30 12:30:40.489560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:40.487963    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87" podStartSLOduration=1.487944916 podStartE2EDuration="1.487944916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:40.482824059 +0000 UTC m=+3644.277708555" watchObservedRunningTime="2026-10-30 12:30:40.487944916 +0000 UTC m=+3644.282829405"
Oct 30 12:30:41.457335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:41.457302    5060 generic.go:356] "Generic (PLEG): container finished" podID="6b748546-4ce6-461f-ae9a-5788fd038e6e" containerID="176125918f676537bb681b30756aa60cf992b79f02ef618d11194e85ce6662e4" exitCode=0
Oct 30 12:30:41.457830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:41.457391    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87" event={"ID":"6b748546-4ce6-461f-ae9a-5788fd038e6e","Type":"ContainerDied","Data":"176125918f676537bb681b30756aa60cf992b79f02ef618d11194e85ce6662e4"}
Oct 30 12:30:41.847145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:41.847040    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:42.675739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.675705    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:42.863591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.863498    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca\") pod \"6b748546-4ce6-461f-ae9a-5788fd038e6e\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") "
Oct 30 12:30:42.863787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.863589    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4\" (UniqueName: \"kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4\") pod \"6b748546-4ce6-461f-ae9a-5788fd038e6e\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") "
Oct 30 12:30:42.863787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.863757    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca\") pod \"6b748546-4ce6-461f-ae9a-5788fd038e6e\" (UID: \"6b748546-4ce6-461f-ae9a-5788fd038e6e\") "
Oct 30 12:30:42.864056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.864031    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca" pod "6b748546-4ce6-461f-ae9a-5788fd038e6e" (UID: "6b748546-4ce6-461f-ae9a-5788fd038e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:42.864511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.864489    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:42.865765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.865740    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca" pod "6b748546-4ce6-461f-ae9a-5788fd038e6e" (UID: "6b748546-4ce6-461f-ae9a-5788fd038e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:30:42.865875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.865836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4" pod "6b748546-4ce6-461f-ae9a-5788fd038e6e" (UID: "6b748546-4ce6-461f-ae9a-5788fd038e6e"). InnerVolumeSpecName "kube-api-access-qxmf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:42.965620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.965581    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b748546-4ce6-461f-ae9a-5788fd038e6e-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:42.965620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:42.965614    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxmf4\" (UniqueName: \"kubernetes.io/projected/6b748546-4ce6-461f-ae9a-5788fd038e6e-kube-api-access-qxmf4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:43.032894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:43.032861    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"]
Oct 30 12:30:43.038118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:43.038089    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e17a68-d2fn4"]
Oct 30 12:30:43.481250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:43.481222    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"
Oct 30 12:30:43.481446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:43.481224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87" event={"ID":"6b748546-4ce6-461f-ae9a-5788fd038e6e","Type":"ContainerDied","Data":"df5b98add957b8a7361b035f04fee1a386c8a05da94498a722d5b1f7cb574374"}
Oct 30 12:30:43.481446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:43.481346    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5b98add957b8a7361b035f04fee1a386c8a05da94498a722d5b1f7cb574374"
Oct 30 12:30:43.661999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:43.661961    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:44.337266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:44.337224    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1881e8f4-396a-447e-b4f6-b274a82910f4" path="/var/lib/kubelet/pods/1881e8f4-396a-447e-b4f6-b274a82910f4/volumes"
Oct 30 12:30:45.085652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:45.085599    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:45.227709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:45.227665    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:45.383263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:45.383157    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"]
Oct 30 12:30:45.384055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:45.384023    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy" containerID="cri-o://688f88e5a5785831a07073230b8c992dd928e6efde749d55defaac7b81943642" gracePeriod=30
Oct 30 12:30:45.453136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:45.453100    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:45.608134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:45.608087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:46.002863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.002818    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"]
Oct 30 12:30:46.003887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.003865    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6b748546-4ce6-461f-ae9a-5788fd038e6e" containerName="revoke-keys"
Oct 30 12:30:46.003949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.003889    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b748546-4ce6-461f-ae9a-5788fd038e6e" containerName="revoke-keys"
Oct 30 12:30:46.004029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.004020    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b748546-4ce6-461f-ae9a-5788fd038e6e" containerName="revoke-keys"
Oct 30 12:30:46.009415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.009388    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.051393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.051351    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:30:46.051629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.051526    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:30:46.064036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.064004    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:30:46.072062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.071968    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"]
Oct 30 12:30:46.081837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.081805    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:30:46.097465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.087255    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"]
Oct 30 12:30:46.097465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.096334    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b97cd30-djscd"]
Oct 30 12:30:46.104636 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.104600    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwz55\" (UniqueName: \"kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.104934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.104913    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.105192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.105174    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.206684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.206636    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.206911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.206840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwz55\" (UniqueName: \"kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.206911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.206898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.207517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.207490    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.209765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.209737    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.230961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.230915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwz55\" (UniqueName: \"kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55\") pod \"maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.340007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.339913    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9" path="/var/lib/kubelet/pods/8afb84a8-3dc9-4e54-b8bd-ae1c9b065ce9/volumes"
Oct 30 12:30:46.357013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.356976    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:46.677930 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:46.677898    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6f0340_b323_4d9b_8d69_f0293d02ff61.slice/crio-c2735dd89e234b9658ab861e822bd6a7dd74e1d1bec48f887bb55a34146dae34 WatchSource:0}: Error finding container c2735dd89e234b9658ab861e822bd6a7dd74e1d1bec48f887bb55a34146dae34: Status 404 returned error can't find the container with id c2735dd89e234b9658ab861e822bd6a7dd74e1d1bec48f887bb55a34146dae34
Oct 30 12:30:46.680648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:46.680620    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"]
Oct 30 12:30:47.346505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:47.346406    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:47.602858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:47.602761    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb" event={"ID":"bb6f0340-b323-4d9b-8d69-f0293d02ff61","Type":"ContainerStarted","Data":"83538fd78290011dd5d84be6d25ac4cf00cb0d2015a75f32e4eb27544378ee97"}
Oct 30 12:30:47.602858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:47.602828    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb" event={"ID":"bb6f0340-b323-4d9b-8d69-f0293d02ff61","Type":"ContainerStarted","Data":"c2735dd89e234b9658ab861e822bd6a7dd74e1d1bec48f887bb55a34146dae34"}
Oct 30 12:30:47.648200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:47.648149    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb" podStartSLOduration=2.648136251 podStartE2EDuration="2.648136251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:47.643777353 +0000 UTC m=+3651.438661849" watchObservedRunningTime="2026-10-30 12:30:47.648136251 +0000 UTC m=+3651.443020746"
Oct 30 12:30:47.765587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:47.765553    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:47.778778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:47.778749    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:48.617074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:48.617021    5060 generic.go:356] "Generic (PLEG): container finished" podID="bb6f0340-b323-4d9b-8d69-f0293d02ff61" containerID="83538fd78290011dd5d84be6d25ac4cf00cb0d2015a75f32e4eb27544378ee97" exitCode=0
Oct 30 12:30:48.617303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:48.617104    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb" event={"ID":"bb6f0340-b323-4d9b-8d69-f0293d02ff61","Type":"ContainerDied","Data":"83538fd78290011dd5d84be6d25ac4cf00cb0d2015a75f32e4eb27544378ee97"}
Oct 30 12:30:48.628948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:48.628915    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:30:49.188896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.188851    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:49.828048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.828022    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:49.859811 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:49.859736    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:30:49.982727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.982692    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55\" (UniqueName: \"kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55\") pod \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") "
Oct 30 12:30:49.982908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.982803    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca\") pod \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") "
Oct 30 12:30:49.982954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.982936    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca\") pod \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\" (UID: \"bb6f0340-b323-4d9b-8d69-f0293d02ff61\") "
Oct 30 12:30:49.983243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.983216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca" pod "bb6f0340-b323-4d9b-8d69-f0293d02ff61" (UID: "bb6f0340-b323-4d9b-8d69-f0293d02ff61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:49.983796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.983770    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:49.985108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.985086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca" pod "bb6f0340-b323-4d9b-8d69-f0293d02ff61" (UID: "bb6f0340-b323-4d9b-8d69-f0293d02ff61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:30:49.985108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:49.985101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55" pod "bb6f0340-b323-4d9b-8d69-f0293d02ff61" (UID: "bb6f0340-b323-4d9b-8d69-f0293d02ff61"). InnerVolumeSpecName "kube-api-access-xwz55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:50.086071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.086031    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwz55\" (UniqueName: \"kubernetes.io/projected/bb6f0340-b323-4d9b-8d69-f0293d02ff61-kube-api-access-xwz55\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.086071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.086072    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6f0340-b323-4d9b-8d69-f0293d02ff61-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.691527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.687965    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"
Oct 30 12:30:50.691527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.690399    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"]
Oct 30 12:30:50.691527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.690464    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb" event={"ID":"bb6f0340-b323-4d9b-8d69-f0293d02ff61","Type":"ContainerDied","Data":"c2735dd89e234b9658ab861e822bd6a7dd74e1d1bec48f887bb55a34146dae34"}
Oct 30 12:30:50.691527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.690503    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2735dd89e234b9658ab861e822bd6a7dd74e1d1bec48f887bb55a34146dae34"
Oct 30 12:30:50.704031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.697071    5060 generic.go:356] "Generic (PLEG): container finished" podID="98605626-4680-4b6c-a7fc-59b59c37d854" containerID="688f88e5a5785831a07073230b8c992dd928e6efde749d55defaac7b81943642" exitCode=0
Oct 30 12:30:50.704031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.697115    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" event={"ID":"98605626-4680-4b6c-a7fc-59b59c37d854","Type":"ContainerDied","Data":"688f88e5a5785831a07073230b8c992dd928e6efde749d55defaac7b81943642"}
Oct 30 12:30:50.704031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.697215    5060 scope.go:122] "RemoveContainer" containerID="83c3140f31d0abdce33683c9ba67b12b2becc7336770d3129f7beb8340cce8a3"
Oct 30 12:30:50.720473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.715672    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:30:50.750281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.744290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855164    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855317    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855350    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855420    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855479    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.856696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.855600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.885478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.881582    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.885478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.881752    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj\") pod \"98605626-4680-4b6c-a7fc-59b59c37d854\" (UID: \"98605626-4680-4b6c-a7fc-59b59c37d854\") "
Oct 30 12:30:50.889716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.889682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.890134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.890112    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.895502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.895438    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:50.966585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.910573    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.966585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.947601    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.966585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.963188    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "kube-api-access-tb8bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:50.999191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.999121    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb8bj\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-kube-api-access-tb8bj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.999371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.999355    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.999516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.999501    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98605626-4680-4b6c-a7fc-59b59c37d854-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.004471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.999932    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.004471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.999956    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.004471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:50.999972    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.012306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.012276    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:30:51.013398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.013358    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:51.016724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.016132    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy" pod "98605626-4680-4b6c-a7fc-59b59c37d854" (UID: "98605626-4680-4b6c-a7fc-59b59c37d854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.026591    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"]
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.027848    5060 cpu_manager.go:414] "containerMap: removing container" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.027868    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.027910    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bb6f0340-b323-4d9b-8d69-f0293d02ff61" containerName="revoke-keys"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.027918    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6f0340-b323-4d9b-8d69-f0293d02ff61" containerName="revoke-keys"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.027974    5060 cpu_manager.go:414] "containerMap: removing container" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.027981    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.028166    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.028184    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" containerName="istio-proxy"
Oct 30 12:30:51.033294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.028201    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6f0340-b323-4d9b-8d69-f0293d02ff61" containerName="revoke-keys"
Oct 30 12:30:51.067471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.034320    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.067471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.066707    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:30:51.067471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.066834    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:30:51.067471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.066919    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:30:51.067857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.067554    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:30:51.075524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.071766    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:51.100826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.095360    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"]
Oct 30 12:30:51.116525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.104964    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9748r\" (UniqueName: \"kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.116525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.105038    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.116525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.105085    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.116525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.105516    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98605626-4680-4b6c-a7fc-59b59c37d854-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.116525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.105587    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98605626-4680-4b6c-a7fc-59b59c37d854-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.116525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.105602    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98605626-4680-4b6c-a7fc-59b59c37d854-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:51.206627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.206592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9748r\" (UniqueName: \"kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.206826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.206644    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.206826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.206673    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.207440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.207413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.209319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.209297    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.232918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.232847    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9748r\" (UniqueName: \"kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r\") pod \"maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.416995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.416964    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:51.710898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.710864    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"
Oct 30 12:30:51.711386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.710906    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q" event={"ID":"98605626-4680-4b6c-a7fc-59b59c37d854","Type":"ContainerDied","Data":"7705b5bcc08b1ffba0aab5bb56221e9945a46ac919821d9cb418452e1660fd40"}
Oct 30 12:30:51.711386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.710971    5060 scope.go:122] "RemoveContainer" containerID="688f88e5a5785831a07073230b8c992dd928e6efde749d55defaac7b81943642"
Oct 30 12:30:51.728170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.728096    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"]
Oct 30 12:30:51.730028 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:30:51.730000    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d47215_f47b_48c1_bd2f_6788f22c0bc0.slice/crio-fd8d66d1cfbc6f5a527340bd5e651fb5ec571533f313d908ca9681caf1dc40f6 WatchSource:0}: Error finding container fd8d66d1cfbc6f5a527340bd5e651fb5ec571533f313d908ca9681caf1dc40f6: Status 404 returned error can't find the container with id fd8d66d1cfbc6f5a527340bd5e651fb5ec571533f313d908ca9681caf1dc40f6
Oct 30 12:30:51.799593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.799557    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"]
Oct 30 12:30:51.807618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:51.807580    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-jmg4q"]
Oct 30 12:30:52.337717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:52.337681    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98605626-4680-4b6c-a7fc-59b59c37d854" path="/var/lib/kubelet/pods/98605626-4680-4b6c-a7fc-59b59c37d854/volumes"
Oct 30 12:30:52.728093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:52.728058    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7" event={"ID":"a6d47215-f47b-48c1-bd2f-6788f22c0bc0","Type":"ContainerStarted","Data":"1fa120590fd0b5527de5a665ea7fc32cb7e5926234518a115fce804c800ee362"}
Oct 30 12:30:52.728627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:52.728110    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7" event={"ID":"a6d47215-f47b-48c1-bd2f-6788f22c0bc0","Type":"ContainerStarted","Data":"fd8d66d1cfbc6f5a527340bd5e651fb5ec571533f313d908ca9681caf1dc40f6"}
Oct 30 12:30:52.774520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:52.774423    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7" podStartSLOduration=2.774408984 podStartE2EDuration="2.774408984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:52.771676222 +0000 UTC m=+3656.566560717" watchObservedRunningTime="2026-10-30 12:30:52.774408984 +0000 UTC m=+3656.569293479"
Oct 30 12:30:52.844335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:52.844295    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:53.633936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:53.633877    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main" containerID="cri-o://df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e" gracePeriod=45
Oct 30 12:30:53.652543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:53.652508    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main" probeResult="failure" output="Get \"https://10.132.1.83:8000/ready\": dial tcp 10.132.1.83:8000: connect: connection refused"
Oct 30 12:30:53.742702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:53.742663    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6d47215-f47b-48c1-bd2f-6788f22c0bc0" containerID="1fa120590fd0b5527de5a665ea7fc32cb7e5926234518a115fce804c800ee362" exitCode=0
Oct 30 12:30:53.743164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:53.742807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7" event={"ID":"a6d47215-f47b-48c1-bd2f-6788f22c0bc0","Type":"ContainerDied","Data":"1fa120590fd0b5527de5a665ea7fc32cb7e5926234518a115fce804c800ee362"}
Oct 30 12:30:53.966023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:53.965996    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:30:54.055572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.055517    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm\") pod \"fd770124-119f-4a0c-8940-f4bff2a64ecf\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") "
Oct 30 12:30:54.055761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.055662    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home\") pod \"fd770124-119f-4a0c-8940-f4bff2a64ecf\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") "
Oct 30 12:30:54.055761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.055721    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache\") pod \"fd770124-119f-4a0c-8940-f4bff2a64ecf\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") "
Oct 30 12:30:54.055873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.055775    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir\") pod \"fd770124-119f-4a0c-8940-f4bff2a64ecf\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") "
Oct 30 12:30:54.055873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.055833    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs\") pod \"fd770124-119f-4a0c-8940-f4bff2a64ecf\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") "
Oct 30 12:30:54.056020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.055976    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home" pod "fd770124-119f-4a0c-8940-f4bff2a64ecf" (UID: "fd770124-119f-4a0c-8940-f4bff2a64ecf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:54.056103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.056021    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8\" (UniqueName: \"kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8\") pod \"fd770124-119f-4a0c-8940-f4bff2a64ecf\" (UID: \"fd770124-119f-4a0c-8940-f4bff2a64ecf\") "
Oct 30 12:30:54.056273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.056188    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir" pod "fd770124-119f-4a0c-8940-f4bff2a64ecf" (UID: "fd770124-119f-4a0c-8940-f4bff2a64ecf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:54.056273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.056213    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache" pod "fd770124-119f-4a0c-8940-f4bff2a64ecf" (UID: "fd770124-119f-4a0c-8940-f4bff2a64ecf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:54.056954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.056766    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:54.056954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.056796    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:54.056954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.056808    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:54.058704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.058682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm" pod "fd770124-119f-4a0c-8940-f4bff2a64ecf" (UID: "fd770124-119f-4a0c-8940-f4bff2a64ecf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:54.058823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.058713    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8" pod "fd770124-119f-4a0c-8940-f4bff2a64ecf" (UID: "fd770124-119f-4a0c-8940-f4bff2a64ecf"). InnerVolumeSpecName "kube-api-access-7xbz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:54.059570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.059548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs" pod "fd770124-119f-4a0c-8940-f4bff2a64ecf" (UID: "fd770124-119f-4a0c-8940-f4bff2a64ecf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:30:54.158218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.158182    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd770124-119f-4a0c-8940-f4bff2a64ecf-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:54.158218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.158215    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbz8\" (UniqueName: \"kubernetes.io/projected/fd770124-119f-4a0c-8940-f4bff2a64ecf-kube-api-access-7xbz8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:54.158218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.158225    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd770124-119f-4a0c-8940-f4bff2a64ecf-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:54.562205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.562169    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:54.705779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.705738    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:54.755062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.755030    5060 generic.go:356] "Generic (PLEG): container finished" podID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerID="df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e" exitCode=0
Oct 30 12:30:54.755589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.755091    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"
Oct 30 12:30:54.755589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.755112    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" event={"ID":"fd770124-119f-4a0c-8940-f4bff2a64ecf","Type":"ContainerDied","Data":"df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e"}
Oct 30 12:30:54.755589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.755171    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn" event={"ID":"fd770124-119f-4a0c-8940-f4bff2a64ecf","Type":"ContainerDied","Data":"fd3aa756cd7b962da8c9a1162a431b3f13fae14702f4e4a2989b3c2f96f61e87"}
Oct 30 12:30:54.755589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.755194    5060 scope.go:122] "RemoveContainer" containerID="df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e"
Oct 30 12:30:54.768412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.768384    5060 scope.go:122] "RemoveContainer" containerID="df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e"
Oct 30 12:30:54.768826 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:30:54.768805    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e\": container with ID starting with df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e not found: ID does not exist" containerID="df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e"
Oct 30 12:30:54.768929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.768833    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e"} err="failed to get container status \"df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e\": rpc error: code = NotFound desc = could not find container \"df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e\": container with ID starting with df6fbf78be0a57afaa2e21cb594fa3f22d3acefba718d596644f0a8008d1658e not found: ID does not exist"
Oct 30 12:30:54.819537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.814856    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"]
Oct 30 12:30:54.831629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.831583    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5cf206/test-model-5cf206-kserve-dcfd96f99-gc4wn"]
Oct 30 12:30:54.978043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:54.977959    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:55.102472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.091039    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca\") pod \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") "
Oct 30 12:30:55.102472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.091156    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r\" (UniqueName: \"kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r\") pod \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") "
Oct 30 12:30:55.102472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.091247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca\") pod \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\" (UID: \"a6d47215-f47b-48c1-bd2f-6788f22c0bc0\") "
Oct 30 12:30:55.102472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.092196    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca" pod "a6d47215-f47b-48c1-bd2f-6788f22c0bc0" (UID: "a6d47215-f47b-48c1-bd2f-6788f22c0bc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:55.102472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.094666    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca" pod "a6d47215-f47b-48c1-bd2f-6788f22c0bc0" (UID: "a6d47215-f47b-48c1-bd2f-6788f22c0bc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:30:55.102472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.094808    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r" pod "a6d47215-f47b-48c1-bd2f-6788f22c0bc0" (UID: "a6d47215-f47b-48c1-bd2f-6788f22c0bc0"). InnerVolumeSpecName "kube-api-access-9748r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:55.193261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.193215    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:55.193261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.193254    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9748r\" (UniqueName: \"kubernetes.io/projected/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-kube-api-access-9748r\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:55.193261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.193265    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d47215-f47b-48c1-bd2f-6788f22c0bc0-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:30:55.238061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.237980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:55.767772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.767742    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"
Oct 30 12:30:55.768177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.767784    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7" event={"ID":"a6d47215-f47b-48c1-bd2f-6788f22c0bc0","Type":"ContainerDied","Data":"fd8d66d1cfbc6f5a527340bd5e651fb5ec571533f313d908ca9681caf1dc40f6"}
Oct 30 12:30:55.768177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:55.767841    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd8d66d1cfbc6f5a527340bd5e651fb5ec571533f313d908ca9681caf1dc40f6"
Oct 30 12:30:56.338526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:56.338487    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" path="/var/lib/kubelet/pods/fd770124-119f-4a0c-8940-f4bff2a64ecf/volumes"
Oct 30 12:30:56.345024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:56.344992    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:56.653064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:56.652976    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:56.797663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:56.797624    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:58.142774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:58.142736    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:30:59.954663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:30:59.954613    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:00.035552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:00.035507    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"]
Oct 30 12:31:00.044753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:00.044719    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3286f98-slfp7"]
Oct 30 12:31:00.338242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:00.338160    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c30fb4b-09fc-4d92-862b-076d098ac88d" path="/var/lib/kubelet/pods/1c30fb4b-09fc-4d92-862b-076d098ac88d/volumes"
Oct 30 12:31:01.413933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:01.413892    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:01.465119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:01.465078    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"]
Oct 30 12:31:01.599944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:01.599869    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:01.737742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:01.737702    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:02.105564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:02.099844    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:02.871065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:02.870980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:02.983320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:02.983280    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"]
Oct 30 12:31:03.014046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:03.014007    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cf206-x6lr7"]
Oct 30 12:31:03.016668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:03.016627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:03.527661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:03.527622    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:03.778770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:03.778676    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:04.338136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.338101    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d47215-f47b-48c1-bd2f-6788f22c0bc0" path="/var/lib/kubelet/pods/a6d47215-f47b-48c1-bd2f-6788f22c0bc0/volumes"
Oct 30 12:31:04.632819 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.632736    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"]
Oct 30 12:31:04.634055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.634031    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6d47215-f47b-48c1-bd2f-6788f22c0bc0" containerName="revoke-keys"
Oct 30 12:31:04.634123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.634060    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d47215-f47b-48c1-bd2f-6788f22c0bc0" containerName="revoke-keys"
Oct 30 12:31:04.634123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.634120    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main"
Oct 30 12:31:04.634199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.634129    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main"
Oct 30 12:31:04.634356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.634335    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd770124-119f-4a0c-8940-f4bff2a64ecf" containerName="main"
Oct 30 12:31:04.634501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.634367    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d47215-f47b-48c1-bd2f-6788f22c0bc0" containerName="revoke-keys"
Oct 30 12:31:04.641650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.641626    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.652135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.652107    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"]
Oct 30 12:31:04.663522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.658789    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-31a6e3\"/\"test-model-31a6e3-kserve-self-signed-certs\""
Oct 30 12:31:04.663522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.658958    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-31a6e3\"/\"default-dockercfg-r2shd\""
Oct 30 12:31:04.670370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.670350    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-31a6e3\"/\"openshift-service-ca.crt\""
Oct 30 12:31:04.670545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.670399    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-31a6e3\"/\"kube-root-ca.crt\""
Oct 30 12:31:04.692530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.692497    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.692728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.692712    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.692835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.692818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.692921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.692907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.693026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.693012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thxwx\" (UniqueName: \"kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.693090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.693054    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.726237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.726195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:04.795476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.795027    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.795476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.795154    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.795476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.795260    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.795476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.795373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thxwx\" (UniqueName: \"kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.795476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.795413    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.795922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.795524    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.797190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.796374    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.798662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.797395    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.798662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.797649    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.801258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.800384    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.811325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.811298    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.824619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.822250    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thxwx\" (UniqueName: \"kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx\") pod \"test-model-31a6e3-kserve-68b6645b77-9v84b\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") " pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:04.990681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:04.988538    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:05.401527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:05.401486    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"]
Oct 30 12:31:05.401954 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:31:05.401559    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3485463_ad00_456d_8b5f_8f6d09ef5c35.slice/crio-d4d08f3d2850751b7d10a8eea7dc8b243535abf0dc3ab4ef4f82b4fffdae7f3b WatchSource:0}: Error finding container d4d08f3d2850751b7d10a8eea7dc8b243535abf0dc3ab4ef4f82b4fffdae7f3b: Status 404 returned error can't find the container with id d4d08f3d2850751b7d10a8eea7dc8b243535abf0dc3ab4ef4f82b4fffdae7f3b
Oct 30 12:31:05.556248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:05.556207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:05.724388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:05.724341    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main" containerID="cri-o://6a5ea18fe13aad135774e7b9610adf2ebd1abb3859984d4cee0c6b633de52a24" gracePeriod=45
Oct 30 12:31:05.729125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:05.729082    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main" probeResult="failure" output="Get \"https://10.132.1.86:8000/ready\": EOF"
Oct 30 12:31:06.042403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.042329    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" event={"ID":"e3485463-ad00-456d-8b5f-8f6d09ef5c35","Type":"ContainerStarted","Data":"02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97"}
Oct 30 12:31:06.042403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.042397    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" event={"ID":"e3485463-ad00-456d-8b5f-8f6d09ef5c35","Type":"ContainerStarted","Data":"d4d08f3d2850751b7d10a8eea7dc8b243535abf0dc3ab4ef4f82b4fffdae7f3b"}
Oct 30 12:31:06.045514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.045430    5060 generic.go:356] "Generic (PLEG): container finished" podID="447ba016-16e1-4916-97b9-ae82475d2557" containerID="6a5ea18fe13aad135774e7b9610adf2ebd1abb3859984d4cee0c6b633de52a24" exitCode=0
Oct 30 12:31:06.045676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.045526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" event={"ID":"447ba016-16e1-4916-97b9-ae82475d2557","Type":"ContainerDied","Data":"6a5ea18fe13aad135774e7b9610adf2ebd1abb3859984d4cee0c6b633de52a24"}
Oct 30 12:31:06.045676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.045568    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm" event={"ID":"447ba016-16e1-4916-97b9-ae82475d2557","Type":"ContainerDied","Data":"d386502bdd24e080dd9efd5f7cb8500300c22f6b79aeb5679a4c2e2623bf9eaf"}
Oct 30 12:31:06.045676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.045583    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d386502bdd24e080dd9efd5f7cb8500300c22f6b79aeb5679a4c2e2623bf9eaf"
Oct 30 12:31:06.049672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.049651    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:31:06.085674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.085614    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" podStartSLOduration=2.085596526 podStartE2EDuration="2.085596526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:06.085039064 +0000 UTC m=+3669.879923557" watchObservedRunningTime="2026-10-30 12:31:06.085596526 +0000 UTC m=+3669.880481032"
Oct 30 12:31:06.134343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134302    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs\" (UniqueName: \"kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs\") pod \"447ba016-16e1-4916-97b9-ae82475d2557\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") "
Oct 30 12:31:06.134559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134391    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir\") pod \"447ba016-16e1-4916-97b9-ae82475d2557\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") "
Oct 30 12:31:06.134559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134410    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache\") pod \"447ba016-16e1-4916-97b9-ae82475d2557\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") "
Oct 30 12:31:06.134749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134607    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home\") pod \"447ba016-16e1-4916-97b9-ae82475d2557\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") "
Oct 30 12:31:06.134749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134646    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc\" (UniqueName: \"kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc\") pod \"447ba016-16e1-4916-97b9-ae82475d2557\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") "
Oct 30 12:31:06.134749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir" pod "447ba016-16e1-4916-97b9-ae82475d2557" (UID: "447ba016-16e1-4916-97b9-ae82475d2557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:06.134749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134709    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache" pod "447ba016-16e1-4916-97b9-ae82475d2557" (UID: "447ba016-16e1-4916-97b9-ae82475d2557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:06.134967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134787    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm\") pod \"447ba016-16e1-4916-97b9-ae82475d2557\" (UID: \"447ba016-16e1-4916-97b9-ae82475d2557\") "
Oct 30 12:31:06.134967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.134865    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home" pod "447ba016-16e1-4916-97b9-ae82475d2557" (UID: "447ba016-16e1-4916-97b9-ae82475d2557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:06.135509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.135482    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:06.135509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.135507    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:06.135698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.135517    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:06.137875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.137844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs" pod "447ba016-16e1-4916-97b9-ae82475d2557" (UID: "447ba016-16e1-4916-97b9-ae82475d2557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:31:06.138003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.137977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm" pod "447ba016-16e1-4916-97b9-ae82475d2557" (UID: "447ba016-16e1-4916-97b9-ae82475d2557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:06.138076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.138027    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc" pod "447ba016-16e1-4916-97b9-ae82475d2557" (UID: "447ba016-16e1-4916-97b9-ae82475d2557"). InnerVolumeSpecName "kube-api-access-d88jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:06.240171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.240122    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447ba016-16e1-4916-97b9-ae82475d2557-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:06.240171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.240169    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88jc\" (UniqueName: \"kubernetes.io/projected/447ba016-16e1-4916-97b9-ae82475d2557-kube-api-access-d88jc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:06.240476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.240186    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447ba016-16e1-4916-97b9-ae82475d2557-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:06.466194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.466137    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z" podUID="29ac4633-3d9a-4d1c-aa26-744a86463c46" containerName="payload-processing" containerID="cri-o://071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53" gracePeriod=25
Oct 30 12:31:06.515128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.514034    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"]
Oct 30 12:31:06.515996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.515966    5060 cpu_manager.go:414] "containerMap: removing container" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main"
Oct 30 12:31:06.515996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.515995    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main"
Oct 30 12:31:06.516279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.516176    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="447ba016-16e1-4916-97b9-ae82475d2557" containerName="main"
Oct 30 12:31:06.536132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.536102    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.548526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.548036    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"]
Oct 30 12:31:06.552021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.551995    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffea51\"/\"default-dockercfg-8j268\""
Oct 30 12:31:06.564306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.564274    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffea51\"/\"kube-root-ca.crt\""
Oct 30 12:31:06.564306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.564297    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffea51\"/\"openshift-service-ca.crt\""
Oct 30 12:31:06.564306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.564274    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffea51\"/\"test-model-ffea51-kserve-self-signed-certs\""
Oct 30 12:31:06.599396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.599302    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:06.650221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.650115    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.650443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.650388    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srh45\" (UniqueName: \"kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.650553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.650499    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.650655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.650615    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.650793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.650768    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.650862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.650815    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.752678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.752645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.752908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.752771    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srh45\" (UniqueName: \"kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.752908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.752817    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.752908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.752896    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.753082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.752992    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.753082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.753032    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.753835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.753809    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.754084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.754060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.754409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.754371    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.755690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.755670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.756809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.756786    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.885843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.876827    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srh45\" (UniqueName: \"kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45\") pod \"test-model-ffea51-kserve-7774879f67-g49r2\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") " pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.918688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.918655    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:06.998475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:06.998137    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:07.031742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.029410    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:31:07.061987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.061947    5060 generic.go:356] "Generic (PLEG): container finished" podID="29ac4633-3d9a-4d1c-aa26-744a86463c46" containerID="071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53" exitCode=0
Oct 30 12:31:07.062165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.061993    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"
Oct 30 12:31:07.062165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.062025    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z" event={"ID":"29ac4633-3d9a-4d1c-aa26-744a86463c46","Type":"ContainerDied","Data":"071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53"}
Oct 30 12:31:07.062165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.062080    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z" event={"ID":"29ac4633-3d9a-4d1c-aa26-744a86463c46","Type":"ContainerDied","Data":"ebe008eec5ba97319361cba897266e20b0356ab62e05ca9231a2715a6f53427a"}
Oct 30 12:31:07.062165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.062109    5060 scope.go:122] "RemoveContainer" containerID="071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53"
Oct 30 12:31:07.062440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.062352    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"
Oct 30 12:31:07.085638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.085604    5060 scope.go:122] "RemoveContainer" containerID="071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53"
Oct 30 12:31:07.087940 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:31:07.087851    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53\": container with ID starting with 071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53 not found: ID does not exist" containerID="071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53"
Oct 30 12:31:07.087940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.087895    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53"} err="failed to get container status \"071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53\": rpc error: code = NotFound desc = could not find container \"071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53\": container with ID starting with 071e21e2625c6053903ff01a9a5aa1023383f8d032f8d93fe3e7db0bbd946a53 not found: ID does not exist"
Oct 30 12:31:07.117629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.117590    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"]
Oct 30 12:31:07.123584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.123553    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9d71fa/test-model-9d71fa-kserve-774989c5ff-6dwtm"]
Oct 30 12:31:07.204845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.204801    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f\" (UniqueName: \"kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f\") pod \"29ac4633-3d9a-4d1c-aa26-744a86463c46\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") "
Oct 30 12:31:07.205158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.205129    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume\") pod \"29ac4633-3d9a-4d1c-aa26-744a86463c46\" (UID: \"29ac4633-3d9a-4d1c-aa26-744a86463c46\") "
Oct 30 12:31:07.206092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.206065    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume" pod "29ac4633-3d9a-4d1c-aa26-744a86463c46" (UID: "29ac4633-3d9a-4d1c-aa26-744a86463c46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:07.209146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.209124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f" pod "29ac4633-3d9a-4d1c-aa26-744a86463c46" (UID: "29ac4633-3d9a-4d1c-aa26-744a86463c46"). InnerVolumeSpecName "kube-api-access-fpt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:07.308034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.307991    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpt8f\" (UniqueName: \"kubernetes.io/projected/29ac4633-3d9a-4d1c-aa26-744a86463c46-kube-api-access-fpt8f\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:07.308034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.308021    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac4633-3d9a-4d1c-aa26-744a86463c46-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:07.356086 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:31:07.356055    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f1c88ae_0a86_4804_8393_9942a3ee2916.slice/crio-bfdfa090a9b8eaf3c513b0aa69f3fcd94b1b80f1aa5585b3027ba4a178aa1d93 WatchSource:0}: Error finding container bfdfa090a9b8eaf3c513b0aa69f3fcd94b1b80f1aa5585b3027ba4a178aa1d93: Status 404 returned error can't find the container with id bfdfa090a9b8eaf3c513b0aa69f3fcd94b1b80f1aa5585b3027ba4a178aa1d93
Oct 30 12:31:07.358051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.358025    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"]
Oct 30 12:31:07.410631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.410602    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"]
Oct 30 12:31:07.419489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.419445    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cf206-56c6bdf6fb-xqd8z"]
Oct 30 12:31:07.435002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:07.434961    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:08.075337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:08.075298    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2" event={"ID":"3f1c88ae-0a86-4804-8393-9942a3ee2916","Type":"ContainerStarted","Data":"842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5"}
Oct 30 12:31:08.075841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:08.075347    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2" event={"ID":"3f1c88ae-0a86-4804-8393-9942a3ee2916","Type":"ContainerStarted","Data":"bfdfa090a9b8eaf3c513b0aa69f3fcd94b1b80f1aa5585b3027ba4a178aa1d93"}
Oct 30 12:31:08.118939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:08.118884    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2" podStartSLOduration=2.118867782 podStartE2EDuration="2.118867782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:08.116931389 +0000 UTC m=+3671.911815907" watchObservedRunningTime="2026-10-30 12:31:08.118867782 +0000 UTC m=+3671.913752275"
Oct 30 12:31:08.337891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:08.337813    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ac4633-3d9a-4d1c-aa26-744a86463c46" path="/var/lib/kubelet/pods/29ac4633-3d9a-4d1c-aa26-744a86463c46/volumes"
Oct 30 12:31:08.338335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:08.338320    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447ba016-16e1-4916-97b9-ae82475d2557" path="/var/lib/kubelet/pods/447ba016-16e1-4916-97b9-ae82475d2557/volumes"
Oct 30 12:31:08.920002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:08.919963    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:09.182125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:09.182028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:10.061477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.056663    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"]
Oct 30 12:31:10.122593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.122559    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3095ccd-48bp4"]
Oct 30 12:31:10.346118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.346022    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:10.348957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.348922    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e05969e-ada7-472a-a845-ecd9973dd9e4" path="/var/lib/kubelet/pods/9e05969e-ada7-472a-a845-ecd9973dd9e4/volumes"
Oct 30 12:31:10.492516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.492472    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:10.657406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.656999    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"]
Oct 30 12:31:10.657406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.657267    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy" containerID="cri-o://00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de" gracePeriod=30
Oct 30 12:31:10.721970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:10.721937    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:11.214809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.214754    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:11.313083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.313049    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"]
Oct 30 12:31:11.313824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.313809    5060 cpu_manager.go:414] "containerMap: removing container" podUID="29ac4633-3d9a-4d1c-aa26-744a86463c46" containerName="payload-processing"
Oct 30 12:31:11.313906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.313825    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ac4633-3d9a-4d1c-aa26-744a86463c46" containerName="payload-processing"
Oct 30 12:31:11.314018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.314008    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ac4633-3d9a-4d1c-aa26-744a86463c46" containerName="payload-processing"
Oct 30 12:31:11.319052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.319032    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.348302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.348256    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:31:11.348778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.348411    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:31:11.352326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.352289    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:31:11.353760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.353734    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"]
Oct 30 12:31:11.354275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.354252    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.354337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.354317    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.354616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.354597    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxdm\" (UniqueName: \"kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.360110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.360068    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:31:11.455712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.455672    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.455896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.455747    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.455896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.455887    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxdm\" (UniqueName: \"kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.456639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.456607    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.458377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.458351    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.480317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.480245    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxdm\" (UniqueName: \"kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm\") pod \"maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.635066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.635026    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:11.954396 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:31:11.954356    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55fce6a_16f4_4143_8778_583aca916dc4.slice/crio-aa6c437a66c530062cde81a2b12732271bf04a83c33ed7f23cd37dd28c4f7b6a WatchSource:0}: Error finding container aa6c437a66c530062cde81a2b12732271bf04a83c33ed7f23cd37dd28c4f7b6a: Status 404 returned error can't find the container with id aa6c437a66c530062cde81a2b12732271bf04a83c33ed7f23cd37dd28c4f7b6a
Oct 30 12:31:11.956726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:11.956704    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"]
Oct 30 12:31:12.161196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:12.161164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf" event={"ID":"f55fce6a-16f4-4143-8778-583aca916dc4","Type":"ContainerStarted","Data":"aa6c437a66c530062cde81a2b12732271bf04a83c33ed7f23cd37dd28c4f7b6a"}
Oct 30 12:31:13.172717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:13.172684    5060 generic.go:356] "Generic (PLEG): container finished" podID="f55fce6a-16f4-4143-8778-583aca916dc4" containerID="79581e6d1ed60e3333bc2d9d3e03be45ec44bb0200744605d2e76755a5daee56" exitCode=0
Oct 30 12:31:13.173168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:13.172714    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf" event={"ID":"f55fce6a-16f4-4143-8778-583aca916dc4","Type":"ContainerDied","Data":"79581e6d1ed60e3333bc2d9d3e03be45ec44bb0200744605d2e76755a5daee56"}
Oct 30 12:31:13.265900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:13.265861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:14.367023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.366998    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:14.510957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.510924    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca\") pod \"f55fce6a-16f4-4143-8778-583aca916dc4\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") "
Oct 30 12:31:14.511193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.510982    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca\") pod \"f55fce6a-16f4-4143-8778-583aca916dc4\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") "
Oct 30 12:31:14.511193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.511035    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm\" (UniqueName: \"kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm\") pod \"f55fce6a-16f4-4143-8778-583aca916dc4\" (UID: \"f55fce6a-16f4-4143-8778-583aca916dc4\") "
Oct 30 12:31:14.511943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.511916    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca" pod "f55fce6a-16f4-4143-8778-583aca916dc4" (UID: "f55fce6a-16f4-4143-8778-583aca916dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:14.514176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.514149    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca" pod "f55fce6a-16f4-4143-8778-583aca916dc4" (UID: "f55fce6a-16f4-4143-8778-583aca916dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:31:14.515272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.514737    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm" pod "f55fce6a-16f4-4143-8778-583aca916dc4" (UID: "f55fce6a-16f4-4143-8778-583aca916dc4"). InnerVolumeSpecName "kube-api-access-gvxdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:14.613899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.613859    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:14.613899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.613896    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55fce6a-16f4-4143-8778-583aca916dc4-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:14.613899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.613907    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxdm\" (UniqueName: \"kubernetes.io/projected/f55fce6a-16f4-4143-8778-583aca916dc4-kube-api-access-gvxdm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:14.988761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:14.988727    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:15.007614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.007575    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:15.084060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.084021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:15.198434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.198392    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"
Oct 30 12:31:15.198434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.198409    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf" event={"ID":"f55fce6a-16f4-4143-8778-583aca916dc4","Type":"ContainerDied","Data":"aa6c437a66c530062cde81a2b12732271bf04a83c33ed7f23cd37dd28c4f7b6a"}
Oct 30 12:31:15.198780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.198535    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6c437a66c530062cde81a2b12732271bf04a83c33ed7f23cd37dd28c4f7b6a"
Oct 30 12:31:15.219384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.219357    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:31:15.964522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:15.964491    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:31:16.050120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050086    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050167    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050198    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050330    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050391    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050523    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050593    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050656    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.050742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.050728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data\") pod \"d61258bd-cad3-4581-a458-1c89606335c5\" (UID: \"d61258bd-cad3-4581-a458-1c89606335c5\") "
Oct 30 12:31:16.051775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.051752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:16.052059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.052014    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:16.052259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.052238    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:16.053202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.052710    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:16.053202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.052993    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:16.055148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.055124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:16.057856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.057831    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:31:16.058020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.057996    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:16.059247 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.059218    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl" pod "d61258bd-cad3-4581-a458-1c89606335c5" (UID: "d61258bd-cad3-4581-a458-1c89606335c5"). InnerVolumeSpecName "kube-api-access-5sfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:16.153120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153077    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153114    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153124    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61258bd-cad3-4581-a458-1c89606335c5-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153133    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sfwl\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-kube-api-access-5sfwl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153141    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61258bd-cad3-4581-a458-1c89606335c5-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153150    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153160    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61258bd-cad3-4581-a458-1c89606335c5-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153174    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.153389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.153182    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61258bd-cad3-4581-a458-1c89606335c5-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:16.209142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.209108    5060 generic.go:356] "Generic (PLEG): container finished" podID="d61258bd-cad3-4581-a458-1c89606335c5" containerID="00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de" exitCode=0
Oct 30 12:31:16.209318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.209158    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"
Oct 30 12:31:16.209318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.209201    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" event={"ID":"d61258bd-cad3-4581-a458-1c89606335c5","Type":"ContainerDied","Data":"00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de"}
Oct 30 12:31:16.209318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.209269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9" event={"ID":"d61258bd-cad3-4581-a458-1c89606335c5","Type":"ContainerDied","Data":"34a355d2d2e0a6583ccd773caa17209f1a70a5b89ab0412dff2e8476450088a0"}
Oct 30 12:31:16.209318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.209297    5060 scope.go:122] "RemoveContainer" containerID="00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de"
Oct 30 12:31:16.225372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.225348    5060 scope.go:122] "RemoveContainer" containerID="00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de"
Oct 30 12:31:16.225698 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:31:16.225669    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de\": container with ID starting with 00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de not found: ID does not exist" containerID="00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de"
Oct 30 12:31:16.225776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.225709    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de"} err="failed to get container status \"00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de\": rpc error: code = NotFound desc = could not find container \"00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de\": container with ID starting with 00d4d5e1874bdaf3d23b7caaefbd4c659fec52242de9e08000da8d70046d66de not found: ID does not exist"
Oct 30 12:31:16.263993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.263965    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"]
Oct 30 12:31:16.269370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.269345    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-5zdw9"]
Oct 30 12:31:16.336782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.336747    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61258bd-cad3-4581-a458-1c89606335c5" path="/var/lib/kubelet/pods/d61258bd-cad3-4581-a458-1c89606335c5/volumes"
Oct 30 12:31:16.844890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.844845    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:16.920103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.920070    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:16.933820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:16.933791    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:17.237185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:17.237156    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:31:18.110802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:18.110763    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"]
Oct 30 12:31:18.111579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:18.111127    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy" containerID="cri-o://54b5996c10ba3d9443b54f1df97c6bca3938539950356e3d7cc91acacd0eafe8" gracePeriod=30
Oct 30 12:31:18.551315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:18.551279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:19.859311 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:31:19.859267    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:31:20.259726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:20.259690    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:21.950329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:21.950286    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:22.092676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:22.092639    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:23.293007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.292974    5060 generic.go:356] "Generic (PLEG): container finished" podID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerID="54b5996c10ba3d9443b54f1df97c6bca3938539950356e3d7cc91acacd0eafe8" exitCode=0
Oct 30 12:31:23.293485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.293052    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" event={"ID":"4de39bfb-a670-43e2-9880-5466d9662ea7","Type":"ContainerDied","Data":"54b5996c10ba3d9443b54f1df97c6bca3938539950356e3d7cc91acacd0eafe8"}
Oct 30 12:31:23.293485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.293202    5060 scope.go:122] "RemoveContainer" containerID="73a54328d8fe3096025b5b03f924a9c47f404217e91508f41245788e2f6ba3a5"
Oct 30 12:31:23.456134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.456103    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:31:23.492372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.492616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492443    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.492616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492558    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.492616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492606    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.492823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492649    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.492823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492677    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:23.492823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492733    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.492823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492760    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.493068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492837    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.493068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.492948    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket\") pod \"4de39bfb-a670-43e2-9880-5466d9662ea7\" (UID: \"4de39bfb-a670-43e2-9880-5466d9662ea7\") "
Oct 30 12:31:23.493509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.493468    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:23.493705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.493682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:23.493791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.493695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:23.493866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.493848    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:23.494327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.494302    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.494443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.494419    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.494559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.494443    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.494559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.494475    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de39bfb-a670-43e2-9880-5466d9662ea7-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.494559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.494488    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.495264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.495231    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:31:23.495570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.495544    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:23.495700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.495568    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "kube-api-access-vkwql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:23.495806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.495788    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy" pod "4de39bfb-a670-43e2-9880-5466d9662ea7" (UID: "4de39bfb-a670-43e2-9880-5466d9662ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:23.595912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.595872    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.595912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.595906    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.595912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.595916    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.596255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.595925    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkwql\" (UniqueName: \"kubernetes.io/projected/4de39bfb-a670-43e2-9880-5466d9662ea7-kube-api-access-vkwql\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:23.832168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:23.832132    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:24.305581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:24.305549    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"
Oct 30 12:31:24.306097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:24.305587    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf" event={"ID":"4de39bfb-a670-43e2-9880-5466d9662ea7","Type":"ContainerDied","Data":"ae2423ef8ddc43f8cfccec97f56b61e6923dbbbd193126a372e5b603c9ee22cf"}
Oct 30 12:31:24.306097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:24.305663    5060 scope.go:122] "RemoveContainer" containerID="54b5996c10ba3d9443b54f1df97c6bca3938539950356e3d7cc91acacd0eafe8"
Oct 30 12:31:24.373621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:24.373587    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"]
Oct 30 12:31:24.386264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:24.386228    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-2j2bf"]
Oct 30 12:31:25.670957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:25.670906    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:26.337743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:26.337712    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" path="/var/lib/kubelet/pods/4de39bfb-a670-43e2-9880-5466d9662ea7/volumes"
Oct 30 12:31:27.355791 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:27.355742    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:29.053952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:29.053919    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:30.878479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:30.873082    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:32.552089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:32.552039    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:34.241314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:34.241269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:35.961124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:35.961085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:37.725402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:37.725364    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:39.413651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:39.413612    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:41.211945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:41.211904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:42.962420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:42.962378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:44.645919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:44.645878    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:46.188683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:46.188614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:46.874742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:46.874686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:46.996157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:46.996119    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"]
Oct 30 12:31:47.007545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:47.007509    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b2407-4n4lb"]
Oct 30 12:31:47.034696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:47.034660    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:47.170231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:47.170099    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:47.525312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:47.525269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:48.337314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:48.337275    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6f0340-b323-4d9b-8d69-f0293d02ff61" path="/var/lib/kubelet/pods/bb6f0340-b323-4d9b-8d69-f0293d02ff61/volumes"
Oct 30 12:31:48.924662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:48.924615    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:49.858052 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:31:49.858011    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:31:50.756441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:50.756405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:52.455003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:52.454958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:53.867737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:53.867696    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:54.013134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:54.013092    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:54.230704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:54.230661    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"]
Oct 30 12:31:54.230970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:54.230877    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy" containerID="cri-o://c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695" gracePeriod=30
Oct 30 12:31:54.329578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:54.329541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:54.487547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:54.479289    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:55.255021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:55.254985    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:55.433328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:55.433293    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:55.533538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:55.525888    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:31:55.614074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:55.613986    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:56.341030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:56.340988    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:58.120657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:58.120615    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:31:59.534165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.534136    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:31:59.688887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.688854    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689039    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689096    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689122    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689170    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:59.689301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689231    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689287    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:59.689518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689355    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689385    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689401    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:59.689757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:59.689757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689618    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data\") pod \"693bff5e-b1b9-4022-9dff-706c5a07bc07\" (UID: \"693bff5e-b1b9-4022-9dff-706c5a07bc07\") "
Oct 30 12:31:59.689949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.689914    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:59.690403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.690380    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.690403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.690404    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.690622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.690415    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.690622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.690425    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.690622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.690437    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693bff5e-b1b9-4022-9dff-706c5a07bc07-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.692026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.691991    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:59.692377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.692359    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "kube-api-access-bc8zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:59.692670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.692647    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:31:59.692775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.692757    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy" pod "693bff5e-b1b9-4022-9dff-706c5a07bc07" (UID: "693bff5e-b1b9-4022-9dff-706c5a07bc07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:31:59.792370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.792277    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.792370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.792320    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.792370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.792336    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693bff5e-b1b9-4022-9dff-706c5a07bc07-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.792370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.792351    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8zq\" (UniqueName: \"kubernetes.io/projected/693bff5e-b1b9-4022-9dff-706c5a07bc07-kube-api-access-bc8zq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:31:59.836769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.836726    5060 generic.go:356] "Generic (PLEG): container finished" podID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerID="c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695" exitCode=0
Oct 30 12:31:59.836930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.836773    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" event={"ID":"693bff5e-b1b9-4022-9dff-706c5a07bc07","Type":"ContainerDied","Data":"c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695"}
Oct 30 12:31:59.836930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.836818    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"
Oct 30 12:31:59.836930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.836831    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9" event={"ID":"693bff5e-b1b9-4022-9dff-706c5a07bc07","Type":"ContainerDied","Data":"23dc59a80e1d670fc00bade24998b26961888d38d8dd30acab36b6e1be63d7b8"}
Oct 30 12:31:59.836930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.836857    5060 scope.go:122] "RemoveContainer" containerID="c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695"
Oct 30 12:31:59.852169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.852146    5060 scope.go:122] "RemoveContainer" containerID="c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"
Oct 30 12:31:59.864600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.864565    5060 scope.go:122] "RemoveContainer" containerID="c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695"
Oct 30 12:31:59.864889 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:31:59.864867    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695\": container with ID starting with c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695 not found: ID does not exist" containerID="c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695"
Oct 30 12:31:59.864986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.864901    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695"} err="failed to get container status \"c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695\": rpc error: code = NotFound desc = could not find container \"c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695\": container with ID starting with c14a9945dfba434c7ee856a29ed88d6d3468180f228e67ea6ff0f6ccd4f53695 not found: ID does not exist"
Oct 30 12:31:59.864986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.864928    5060 scope.go:122] "RemoveContainer" containerID="c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"
Oct 30 12:31:59.865205 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:31:59.865184    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67\": container with ID starting with c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67 not found: ID does not exist" containerID="c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"
Oct 30 12:31:59.865260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.865213    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67"} err="failed to get container status \"c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67\": rpc error: code = NotFound desc = could not find container \"c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67\": container with ID starting with c00779b5eba81a88c82adb9b9b5cb4370435e5de7486c26e3faedcbf78d7dc67 not found: ID does not exist"
Oct 30 12:31:59.894254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.894220    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"]
Oct 30 12:31:59.906341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.906314    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9b2407-openshift-default-5499469645-bgjv9"]
Oct 30 12:31:59.978376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:31:59.973477    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:00.342945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:00.342903    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" path="/var/lib/kubelet/pods/693bff5e-b1b9-4022-9dff-706c5a07bc07/volumes"
Oct 30 12:32:01.639189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:01.639154    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:02.753934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.753904    5060 scope.go:122] "RemoveContainer" containerID="c0a5a7c7fd662eda909f1b1e731f586e0c7e8f1140ff4416359b53dd9afc8d34"
Oct 30 12:32:02.765392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.765362    5060 scope.go:122] "RemoveContainer" containerID="0b84dff20e3df3d709b9e373345a0819e65d36569ca9055251ebefffb68511ba"
Oct 30 12:32:02.776029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.776006    5060 scope.go:122] "RemoveContainer" containerID="aa82eb9b92d7d1dae60712d0d57a2516c4f770ee34478af65c282e38028d8b52"
Oct 30 12:32:02.789694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.789649    5060 scope.go:122] "RemoveContainer" containerID="7678d5dc1fde6ccf023de5ee626151cf8550a0dffc48e5f3f8df8f2408766d2b"
Oct 30 12:32:02.802607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.802583    5060 scope.go:122] "RemoveContainer" containerID="4266152fbce078c800b7cd4e322cc77b0b0085ea3f8068256a794e5884ddd084"
Oct 30 12:32:02.817950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.817923    5060 scope.go:122] "RemoveContainer" containerID="6a5ea18fe13aad135774e7b9610adf2ebd1abb3859984d4cee0c6b633de52a24"
Oct 30 12:32:02.829928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.829907    5060 scope.go:122] "RemoveContainer" containerID="3cca9714d2b2b597f3f312609ff222da6558992555a93b8a2c13723e147ed523"
Oct 30 12:32:02.841693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:02.841662    5060 scope.go:122] "RemoveContainer" containerID="fcd98497d835cf5cd190957060d7b9ccbb4b87f7ab06f59241049fda7d9263bf"
Oct 30 12:32:03.306270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:03.306234    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:05.147490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:05.147432    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:06.608971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:06.608883    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:07.240327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:07.240295    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:07.387996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:07.387962    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"]
Oct 30 12:32:07.412018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:07.411980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:07.419482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:07.419431    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d71fa-flpnf"]
Oct 30 12:32:07.556509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:07.556411    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:07.931561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:07.931478    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:08.336683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:08.336649    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55fce6a-16f4-4143-8778-583aca916dc4" path="/var/lib/kubelet/pods/f55fce6a-16f4-4143-8778-583aca916dc4/volumes"
Oct 30 12:32:09.265651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:09.265594    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:11.052224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:11.052186    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:12.803827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:12.803775    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:14.018372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.018332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:14.171223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.171176    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:14.316295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.315426    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"]
Oct 30 12:32:14.316295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.315770    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" containerID="cri-o://71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e" gracePeriod=30
Oct 30 12:32:14.327033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.326999    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"]
Oct 30 12:32:14.327222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.327187    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" containerID="cri-o://e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d" gracePeriod=30
Oct 30 12:32:14.379024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.378985    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:14.785059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:14.785018    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:15.379133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:15.379097    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:15.530550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:15.530517    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:15.671388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:15.671306    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:15.840169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:15.840136    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:32:15.840350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:15.840195    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:32:16.265975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:16.265936    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:16.525004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:16.524901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:18.338506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:18.338431    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:18.363921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:18.363887    5060 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:32:18.364094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:18.363943    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:32:19.688052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.688015    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:32:19.690060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.690032    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:32:19.757060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757028    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.757260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757072    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.757260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757110    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.757260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757131    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.757260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.757260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757189    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.757615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757379    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.757615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757489    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.757615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757526    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.757615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757557    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.757615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757594    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.757890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757678    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.757890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757683    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.757890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757748    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.757890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757800    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.758134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757898    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.758134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.757903    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.758134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758010    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.758134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758058    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.758134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758110    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:19.758134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758117    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.758493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758164    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.758493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758330    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token\") pod \"5f872375-b24f-47cf-984e-06b1e94b9b2c\" (UID: \"5f872375-b24f-47cf-984e-06b1e94b9b2c\") "
Oct 30 12:32:19.758493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758376    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data\") pod \"3403dde0-012d-447a-a9a2-0516569d1770\" (UID: \"3403dde0-012d-447a-a9a2-0516569d1770\") "
Oct 30 12:32:19.758493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.758406    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:19.759399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.759374    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.759599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.759582    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.759717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.759702    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f872375-b24f-47cf-984e-06b1e94b9b2c-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.759833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.759820    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.759936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.759924    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.760095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760077    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.760209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760195    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3403dde0-012d-447a-a9a2-0516569d1770-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.760301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760169    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.760383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760361    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.760383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760369    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.760551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760521    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.760651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.760629    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.761115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.761093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "kube-api-access-chx4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.761224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.761110    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.761224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.761207    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:32:19.761355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.761328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:32:19.762532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.762511    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token" pod "5f872375-b24f-47cf-984e-06b1e94b9b2c" (UID: "5f872375-b24f-47cf-984e-06b1e94b9b2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.762927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.762907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9" pod "3403dde0-012d-447a-a9a2-0516569d1770" (UID: "3403dde0-012d-447a-a9a2-0516569d1770"). InnerVolumeSpecName "kube-api-access-tpcv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.859541 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:32:19.859506    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:32:19.861582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861563    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861584    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861592    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861603    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861611    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3403dde0-012d-447a-a9a2-0516569d1770-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861618    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861627    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcv9\" (UniqueName: \"kubernetes.io/projected/3403dde0-012d-447a-a9a2-0516569d1770-kube-api-access-tpcv9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861638    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chx4g\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-kube-api-access-chx4g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861648    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f872375-b24f-47cf-984e-06b1e94b9b2c-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861656    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f872375-b24f-47cf-984e-06b1e94b9b2c-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.861710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:19.861663    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3403dde0-012d-447a-a9a2-0516569d1770-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:32:20.128288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.128254    5060 generic.go:356] "Generic (PLEG): container finished" podID="3403dde0-012d-447a-a9a2-0516569d1770" containerID="71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e" exitCode=0
Oct 30 12:32:20.128542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.128338    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"
Oct 30 12:32:20.128542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.128371    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" event={"ID":"3403dde0-012d-447a-a9a2-0516569d1770","Type":"ContainerDied","Data":"71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e"}
Oct 30 12:32:20.128542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.128432    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4" event={"ID":"3403dde0-012d-447a-a9a2-0516569d1770","Type":"ContainerDied","Data":"19872d3c213f46f42c865390f7f6f2e5e97cbe443f544a7c93f86b8561a81a5a"}
Oct 30 12:32:20.128542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.128483    5060 scope.go:122] "RemoveContainer" containerID="71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e"
Oct 30 12:32:20.130206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.130182    5060 generic.go:356] "Generic (PLEG): container finished" podID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerID="e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d" exitCode=0
Oct 30 12:32:20.130318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.130226    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"
Oct 30 12:32:20.130318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.130269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" event={"ID":"5f872375-b24f-47cf-984e-06b1e94b9b2c","Type":"ContainerDied","Data":"e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d"}
Oct 30 12:32:20.130318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.130315    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt" event={"ID":"5f872375-b24f-47cf-984e-06b1e94b9b2c","Type":"ContainerDied","Data":"8d3796516c24d93c2d4f6e92c53cb5a0fe4657f4450d261d5efcd94cbb09632b"}
Oct 30 12:32:20.141521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.141497    5060 scope.go:122] "RemoveContainer" containerID="2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"
Oct 30 12:32:20.152995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.152969    5060 scope.go:122] "RemoveContainer" containerID="71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e"
Oct 30 12:32:20.153313 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:32:20.153283    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e\": container with ID starting with 71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e not found: ID does not exist" containerID="71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e"
Oct 30 12:32:20.153423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.153322    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e"} err="failed to get container status \"71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e\": rpc error: code = NotFound desc = could not find container \"71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e\": container with ID starting with 71d02573a1a8bba677dff71ac5132af56ad38aa890e4c70a3f3612deb7e43a6e not found: ID does not exist"
Oct 30 12:32:20.153423 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.153343    5060 scope.go:122] "RemoveContainer" containerID="2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"
Oct 30 12:32:20.153640 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:32:20.153620    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80\": container with ID starting with 2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80 not found: ID does not exist" containerID="2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"
Oct 30 12:32:20.153686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.153647    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80"} err="failed to get container status \"2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80\": rpc error: code = NotFound desc = could not find container \"2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80\": container with ID starting with 2453bd468d1bc989c478688148916b00780a58b3868be1252f381f43b151ba80 not found: ID does not exist"
Oct 30 12:32:20.153686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.153663    5060 scope.go:122] "RemoveContainer" containerID="e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d"
Oct 30 12:32:20.163982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.163699    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:20.198929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.198877    5060 scope.go:122] "RemoveContainer" containerID="e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d"
Oct 30 12:32:20.199429 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:32:20.199401    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d\": container with ID starting with e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d not found: ID does not exist" containerID="e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d"
Oct 30 12:32:20.199707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.199508    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d"} err="failed to get container status \"e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d\": rpc error: code = NotFound desc = could not find container \"e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d\": container with ID starting with e4010ca19630699f9c8095a26611c96441867f19eece6ab9a09a166adbb5a43d not found: ID does not exist"
Oct 30 12:32:20.201809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.201781    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"]
Oct 30 12:32:20.207492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.207446    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-lt8g4"]
Oct 30 12:32:20.232104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.231486    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"]
Oct 30 12:32:20.236610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.236571    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v9jnt"]
Oct 30 12:32:20.338176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.338142    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3403dde0-012d-447a-a9a2-0516569d1770" path="/var/lib/kubelet/pods/3403dde0-012d-447a-a9a2-0516569d1770/volumes"
Oct 30 12:32:20.338774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:20.338756    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" path="/var/lib/kubelet/pods/5f872375-b24f-47cf-984e-06b1e94b9b2c/volumes"
Oct 30 12:32:21.956897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:21.956864    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:23.968499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:23.968435    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:25.860400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:25.860362    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:27.608999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:27.608958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:29.358355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:29.358317    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:31.051475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:31.048623    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:32.772126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:32.768211    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:34.530402 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:34.530366    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:36.203034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:36.202994    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:38.060909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:38.060842    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:39.752044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:39.752005    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:41.440489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:41.440427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:42.571330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:42.571284    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:43.157108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:43.157068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:44.831373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:44.831339    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:46.545951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:46.545915    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:47.073814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:47.073781    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:47.219307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:47.219261    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:48.366716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.366678    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:48.724882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.724838    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"]
Oct 30 12:32:48.725897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725868    5060 cpu_manager.go:414] "containerMap: removing container" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy"
Oct 30 12:32:48.725897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725894    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725912    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f55fce6a-16f4-4143-8778-583aca916dc4" containerName="revoke-keys"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725917    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55fce6a-16f4-4143-8778-583aca916dc4" containerName="revoke-keys"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725927    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725932    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725940    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725945    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725956    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725960    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725970    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725974    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725982    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.725986    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726003    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy"
Oct 30 12:32:48.726065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726008    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726128    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55fce6a-16f4-4143-8778-583aca916dc4" containerName="revoke-keys"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726137    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726143    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de39bfb-a670-43e2-9880-5466d9662ea7" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726150    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726156    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61258bd-cad3-4581-a458-1c89606335c5" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726162    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3403dde0-012d-447a-a9a2-0516569d1770" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726171    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f872375-b24f-47cf-984e-06b1e94b9b2c" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726179    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy"
Oct 30 12:32:48.726588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.726187    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy"
Oct 30 12:32:48.730110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.730089    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.765485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.765417    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"]
Oct 30 12:32:48.788848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.788797    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.788888    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.788981    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.789017    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.789072    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlh5q\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.789111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.789156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.789180    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.789427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.789245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.804826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.804786    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:48.890678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890635    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.890884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890708    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.890884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890788    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.890884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890817    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.890884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890860    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlh5q\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.890968    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.891030    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.891130    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.891397    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.891487    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.891655    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.891894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.891845    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.893682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.893659    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.893682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.893672    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.914936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.914905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlh5q\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:48.915114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:48.914905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fwdcl\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:49.046728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.046649    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:49.373445 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:32:49.373415    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3b99f3_488c_41a0_a86b_9615f6d6aca7.slice/crio-3c395f582cfcc26fefceb960cef51abfc13ecdcb5f5bc2df6f8ec632d60a165b WatchSource:0}: Error finding container 3c395f582cfcc26fefceb960cef51abfc13ecdcb5f5bc2df6f8ec632d60a165b: Status 404 returned error can't find the container with id 3c395f582cfcc26fefceb960cef51abfc13ecdcb5f5bc2df6f8ec632d60a165b
Oct 30 12:32:49.375486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.375438    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"]
Oct 30 12:32:49.376951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.376915    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:32:49.377052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.376982    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:32:49.377052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.377016    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:32:49.485109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.485064    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" event={"ID":"6f3b99f3-488c-41a0-a86b-9615f6d6aca7","Type":"ContainerStarted","Data":"619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db"}
Oct 30 12:32:49.485302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.485134    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" event={"ID":"6f3b99f3-488c-41a0-a86b-9615f6d6aca7","Type":"ContainerStarted","Data":"3c395f582cfcc26fefceb960cef51abfc13ecdcb5f5bc2df6f8ec632d60a165b"}
Oct 30 12:32:49.533445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:49.533387    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podStartSLOduration=1.533372417 podStartE2EDuration="1.533372417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:49.530878285 +0000 UTC m=+3773.325762793" watchObservedRunningTime="2026-10-30 12:32:49.533372417 +0000 UTC m=+3773.328256977"
Oct 30 12:32:49.860641 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:32:49.860592    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:32:50.047180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:50.047143    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:50.048563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:50.048537    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:50.048711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:50.048591    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:50.134179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:50.134097    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:51.047702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:51.047667    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:51.048119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:51.047720    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:51.917119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:51.917078    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:52.048026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:52.047989    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:52.048437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:52.048048    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:53.047076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:53.047036    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:53.047270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:53.047093    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:53.825126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:53.825094    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:54.047891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:54.047846    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:54.048221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:54.048188    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:55.047568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:55.047531    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:55.047962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:55.047599    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:55.501482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:55.497784    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:55.918944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:55.918840    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:56.047615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:56.047580    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:56.048023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:56.047635    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:57.047843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:57.047797    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 12:32:57.048257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:57.047868    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 12:32:57.655298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:57.655248    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:58.613901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:58.613864    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:58.615289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:58.615264    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:58.616234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:58.616213    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:32:59.110310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.110274    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"]
Oct 30 12:32:59.111524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.111501    5060 cpu_manager.go:414] "containerMap: removing container" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy"
Oct 30 12:32:59.111630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.111526    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693bff5e-b1b9-4022-9dff-706c5a07bc07" containerName="istio-proxy"
Oct 30 12:32:59.116466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.116430    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.131337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.131303    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"]
Oct 30 12:32:59.131550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.131436    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:32:59.131746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.131728    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:32:59.131841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.131784    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:32:59.131908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.131861    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:32:59.243699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.243661    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.243879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.243719    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.243879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.243833    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxn46\" (UniqueName: \"kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.345047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.345011    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.345278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.345062    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.345278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.345110    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxn46\" (UniqueName: \"kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.345835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.345809    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.347645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.347621    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.355290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.355258    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:32:59.379555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.379425    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxn46\" (UniqueName: \"kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46\") pod \"maas-api-revoke-sub-c1d70d0f-8n9gd\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.434970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.434942    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:32:59.753106 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:32:59.753075    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c157d9_e3cb_4b87_b145_268bfca0a860.slice/crio-9c7226a548c58911abe4eee0a19b4883a211c6fc32ad51b1e2fdf824b8710be1 WatchSource:0}: Error finding container 9c7226a548c58911abe4eee0a19b4883a211c6fc32ad51b1e2fdf824b8710be1: Status 404 returned error can't find the container with id 9c7226a548c58911abe4eee0a19b4883a211c6fc32ad51b1e2fdf824b8710be1
Oct 30 12:32:59.758104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:32:59.758078    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"]
Oct 30 12:33:00.636739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:00.636700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd" event={"ID":"56c157d9-e3cb-4b87-b145-268bfca0a860","Type":"ContainerStarted","Data":"15651cdd2d8bf970e88f4a0249250f18cb0e911a92eb19d4b47191e78d587605"}
Oct 30 12:33:00.636938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:00.636759    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd" event={"ID":"56c157d9-e3cb-4b87-b145-268bfca0a860","Type":"ContainerStarted","Data":"9c7226a548c58911abe4eee0a19b4883a211c6fc32ad51b1e2fdf824b8710be1"}
Oct 30 12:33:00.681529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:00.681444    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd" podStartSLOduration=1.681425626 podStartE2EDuration="1.681425626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:00.677288588 +0000 UTC m=+3784.472173083" watchObservedRunningTime="2026-10-30 12:33:00.681425626 +0000 UTC m=+3784.476310121"
Oct 30 12:33:01.160434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:01.160320    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:01.648306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:01.648265    5060 generic.go:356] "Generic (PLEG): container finished" podID="56c157d9-e3cb-4b87-b145-268bfca0a860" containerID="15651cdd2d8bf970e88f4a0249250f18cb0e911a92eb19d4b47191e78d587605" exitCode=0
Oct 30 12:33:01.648549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:01.648349    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd" event={"ID":"56c157d9-e3cb-4b87-b145-268bfca0a860","Type":"ContainerDied","Data":"15651cdd2d8bf970e88f4a0249250f18cb0e911a92eb19d4b47191e78d587605"}
Oct 30 12:33:02.842873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.842840    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:02.888940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.884773    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:33:02.920934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.920842    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca\") pod \"56c157d9-e3cb-4b87-b145-268bfca0a860\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") "
Oct 30 12:33:02.921102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.921043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca\") pod \"56c157d9-e3cb-4b87-b145-268bfca0a860\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") "
Oct 30 12:33:02.921102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.921096    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46\" (UniqueName: \"kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46\") pod \"56c157d9-e3cb-4b87-b145-268bfca0a860\" (UID: \"56c157d9-e3cb-4b87-b145-268bfca0a860\") "
Oct 30 12:33:02.921464 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.921421    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca" pod "56c157d9-e3cb-4b87-b145-268bfca0a860" (UID: "56c157d9-e3cb-4b87-b145-268bfca0a860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:02.921783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.921766    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:02.924581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.924542    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca" pod "56c157d9-e3cb-4b87-b145-268bfca0a860" (UID: "56c157d9-e3cb-4b87-b145-268bfca0a860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:02.924683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.924581    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46" pod "56c157d9-e3cb-4b87-b145-268bfca0a860" (UID: "56c157d9-e3cb-4b87-b145-268bfca0a860"). InnerVolumeSpecName "kube-api-access-vxn46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:02.951934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:02.951900    5060 scope.go:122] "RemoveContainer" containerID="ce9ff964d2715cde1058522fbda2570c6a6ceb8ea8e095c7aece0f3c4ba9b63d"
Oct 30 12:33:03.023272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:03.023241    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxn46\" (UniqueName: \"kubernetes.io/projected/56c157d9-e3cb-4b87-b145-268bfca0a860-kube-api-access-vxn46\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:03.023272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:03.023270    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c157d9-e3cb-4b87-b145-268bfca0a860-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:03.669671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:03.669634    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd" event={"ID":"56c157d9-e3cb-4b87-b145-268bfca0a860","Type":"ContainerDied","Data":"9c7226a548c58911abe4eee0a19b4883a211c6fc32ad51b1e2fdf824b8710be1"}
Oct 30 12:33:03.669856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:03.669689    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c7226a548c58911abe4eee0a19b4883a211c6fc32ad51b1e2fdf824b8710be1"
Oct 30 12:33:03.669856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:03.669645    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"
Oct 30 12:33:04.515156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:04.515116    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:06.260797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:06.260756    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:07.996987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:07.996946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:09.523720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:09.523678    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:10.808914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.808872    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"]
Oct 30 12:33:10.809607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.809590    5060 cpu_manager.go:414] "containerMap: removing container" podUID="56c157d9-e3cb-4b87-b145-268bfca0a860" containerName="revoke-keys"
Oct 30 12:33:10.809607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.809607    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c157d9-e3cb-4b87-b145-268bfca0a860" containerName="revoke-keys"
Oct 30 12:33:10.809772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.809763    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c157d9-e3cb-4b87-b145-268bfca0a860" containerName="revoke-keys"
Oct 30 12:33:10.815313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.815291    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:10.832506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.832475    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"]
Oct 30 12:33:10.838362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.838334    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:33:10.838362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.838348    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:33:10.838632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.838522    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:33:10.842378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.842354    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:33:10.935859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.935826    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:10.975896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.975859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:10.976082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.975954    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2zs\" (UniqueName: \"kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:10.976162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:10.976148    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.077380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.077281    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.077642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.077413    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.077642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.077517    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2zs\" (UniqueName: \"kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.078257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.078229    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.080021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.079994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.101089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.101062    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2zs\" (UniqueName: \"kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs\") pod \"maas-api-revoke-sub-dba98891-cg5sm\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.132709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.132673    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:11.428934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.428840    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:11.478044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.478011    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"]
Oct 30 12:33:11.483178 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:33:11.483149    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de5723b_36f4_40cb_946c_99b3f1939305.slice/crio-e52279334d9548e985657ce1439f97299f0405665f524ffe654d248b165dba49 WatchSource:0}: Error finding container e52279334d9548e985657ce1439f97299f0405665f524ffe654d248b165dba49: Status 404 returned error can't find the container with id e52279334d9548e985657ce1439f97299f0405665f524ffe654d248b165dba49
Oct 30 12:33:11.817794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:11.817757    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm" event={"ID":"9de5723b-36f4-40cb-946c-99b3f1939305","Type":"ContainerStarted","Data":"e52279334d9548e985657ce1439f97299f0405665f524ffe654d248b165dba49"}
Oct 30 12:33:12.829212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:12.829171    5060 generic.go:356] "Generic (PLEG): container finished" podID="9de5723b-36f4-40cb-946c-99b3f1939305" containerID="77244d96e46de8d0fbb4a7c33b90481f121b5ab60d59f8c3e909a15408e9808d" exitCode=0
Oct 30 12:33:12.829662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:12.829258    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm" event={"ID":"9de5723b-36f4-40cb-946c-99b3f1939305","Type":"ContainerDied","Data":"77244d96e46de8d0fbb4a7c33b90481f121b5ab60d59f8c3e909a15408e9808d"}
Oct 30 12:33:13.257304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:13.257268    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:14.023181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.023142    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:14.125909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.125828    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs\" (UniqueName: \"kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs\") pod \"9de5723b-36f4-40cb-946c-99b3f1939305\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") "
Oct 30 12:33:14.125909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.125866    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca\") pod \"9de5723b-36f4-40cb-946c-99b3f1939305\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") "
Oct 30 12:33:14.126177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.125914    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca\") pod \"9de5723b-36f4-40cb-946c-99b3f1939305\" (UID: \"9de5723b-36f4-40cb-946c-99b3f1939305\") "
Oct 30 12:33:14.126346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.126319    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca" pod "9de5723b-36f4-40cb-946c-99b3f1939305" (UID: "9de5723b-36f4-40cb-946c-99b3f1939305"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:14.126717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.126655    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:14.128239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.128214    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca" pod "9de5723b-36f4-40cb-946c-99b3f1939305" (UID: "9de5723b-36f4-40cb-946c-99b3f1939305"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:14.128445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.128421    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs" pod "9de5723b-36f4-40cb-946c-99b3f1939305" (UID: "9de5723b-36f4-40cb-946c-99b3f1939305"). InnerVolumeSpecName "kube-api-access-db2zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:14.228072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.228036    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db2zs\" (UniqueName: \"kubernetes.io/projected/9de5723b-36f4-40cb-946c-99b3f1939305-kube-api-access-db2zs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:14.228072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.228075    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5723b-36f4-40cb-946c-99b3f1939305-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:14.850606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.850573    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"
Oct 30 12:33:14.850763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.850610    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm" event={"ID":"9de5723b-36f4-40cb-946c-99b3f1939305","Type":"ContainerDied","Data":"e52279334d9548e985657ce1439f97299f0405665f524ffe654d248b165dba49"}
Oct 30 12:33:14.850763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.850663    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e52279334d9548e985657ce1439f97299f0405665f524ffe654d248b165dba49"
Oct 30 12:33:14.946316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:14.946265    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:15.604968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:15.604935    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:16.690200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:16.690160    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:18.376090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:18.376057    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:19.861405 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:33:19.861367    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:33:20.054740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:20.054701    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:21.809974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:21.809925    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"]
Oct 30 12:33:21.856644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:21.856551    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:21.997636 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:21.997602    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:22.367528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.367494    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"]
Oct 30 12:33:22.368254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.368237    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9de5723b-36f4-40cb-946c-99b3f1939305" containerName="revoke-keys"
Oct 30 12:33:22.368319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.368256    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de5723b-36f4-40cb-946c-99b3f1939305" containerName="revoke-keys"
Oct 30 12:33:22.368399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.368390    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de5723b-36f4-40cb-946c-99b3f1939305" containerName="revoke-keys"
Oct 30 12:33:22.372345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.372323    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.389913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.389885    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"]
Oct 30 12:33:22.396190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.396127    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:33:22.396730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.396702    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:33:22.396899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.396750    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:33:22.397003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.396787    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:33:22.453782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.453747    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.453985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.453818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgx5m\" (UniqueName: \"kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.453985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.453950    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.554993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.554958    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.555205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.555054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgx5m\" (UniqueName: \"kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.555205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.555105    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.555760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.555738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.557713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.557688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.578664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.578637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgx5m\" (UniqueName: \"kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m\") pod \"maas-api-revoke-sub-57d7505b-g5lbk\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:22.689724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:22.689634    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:23.004179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:23.004147    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"]
Oct 30 12:33:23.623955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:23.623918    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:24.018384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:24.018344    5060 generic.go:356] "Generic (PLEG): container finished" podID="732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" containerID="7971f57dd8ad86f44c30cc50f57914c553156dc72d39f7f29a632fa0a77387b5" exitCode=0
Oct 30 12:33:24.018939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:24.018502    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk" event={"ID":"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3","Type":"ContainerDied","Data":"7971f57dd8ad86f44c30cc50f57914c553156dc72d39f7f29a632fa0a77387b5"}
Oct 30 12:33:24.018939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:24.018554    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk" event={"ID":"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3","Type":"ContainerStarted","Data":"c35584f066ea37ec4708ee31d35b5e1a503408173b4c3ba2753c888e6ce9bc08"}
Oct 30 12:33:25.253155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.253119    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:25.298761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.298674    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:25.406471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.406412    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca\") pod \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") "
Oct 30 12:33:25.406707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.406579    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca\") pod \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") "
Oct 30 12:33:25.406849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.406828    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m\" (UniqueName: \"kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m\") pod \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\" (UID: \"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3\") "
Oct 30 12:33:25.406918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.406891    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca" pod "732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" (UID: "732cfea0-b4ca-43a6-9be1-b8f6a6a51be3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:25.407749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.407725    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:25.409068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.409038    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca" pod "732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" (UID: "732cfea0-b4ca-43a6-9be1-b8f6a6a51be3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:25.409223 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.409199    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m" pod "732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" (UID: "732cfea0-b4ca-43a6-9be1-b8f6a6a51be3"). InnerVolumeSpecName "kube-api-access-rgx5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:25.509165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.509124    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:25.509165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:25.509161    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgx5m\" (UniqueName: \"kubernetes.io/projected/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3-kube-api-access-rgx5m\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:26.037266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:26.037238    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"
Oct 30 12:33:26.037473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:26.037265    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk" event={"ID":"732cfea0-b4ca-43a6-9be1-b8f6a6a51be3","Type":"ContainerDied","Data":"c35584f066ea37ec4708ee31d35b5e1a503408173b4c3ba2753c888e6ce9bc08"}
Oct 30 12:33:26.037473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:26.037317    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c35584f066ea37ec4708ee31d35b5e1a503408173b4c3ba2753c888e6ce9bc08"
Oct 30 12:33:27.064703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:27.064627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:28.830046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:28.830003    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:31.139475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:31.136918    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:32.984978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:32.984937    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:34.842098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:34.842063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:36.544971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:36.544884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:36.848613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:36.848492    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" podUID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" containerName="main" containerID="cri-o://1bf1ab6f5b875d63f5e15a31cef9610a7fa63a75670b54133f396d64b248ef4e" gracePeriod=45
Oct 30 12:33:37.149542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.149504    5060 generic.go:356] "Generic (PLEG): container finished" podID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" containerID="1bf1ab6f5b875d63f5e15a31cef9610a7fa63a75670b54133f396d64b248ef4e" exitCode=0
Oct 30 12:33:37.149677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.149560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" event={"ID":"c94dfe3c-1456-4ba2-9c76-8f07f9296871","Type":"ContainerDied","Data":"1bf1ab6f5b875d63f5e15a31cef9610a7fa63a75670b54133f396d64b248ef4e"}
Oct 30 12:33:37.149677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.149613    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj" event={"ID":"c94dfe3c-1456-4ba2-9c76-8f07f9296871","Type":"ContainerDied","Data":"900c337f24f2c92c2e523d78cb38407f02f3478637da6d34834bbad07854f076"}
Oct 30 12:33:37.149677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.149628    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="900c337f24f2c92c2e523d78cb38407f02f3478637da6d34834bbad07854f076"
Oct 30 12:33:37.172257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.172233    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:33:37.256975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.256942    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home\") pod \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") "
Oct 30 12:33:37.257224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257066    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache\") pod \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") "
Oct 30 12:33:37.257224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257125    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d\" (UniqueName: \"kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d\") pod \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") "
Oct 30 12:33:37.257224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257178    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir\") pod \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") "
Oct 30 12:33:37.257789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257261    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs\" (UniqueName: \"kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs\") pod \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") "
Oct 30 12:33:37.257789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257275    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home" pod "c94dfe3c-1456-4ba2-9c76-8f07f9296871" (UID: "c94dfe3c-1456-4ba2-9c76-8f07f9296871"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:37.257789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm\") pod \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\" (UID: \"c94dfe3c-1456-4ba2-9c76-8f07f9296871\") "
Oct 30 12:33:37.257789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257510    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir" pod "c94dfe3c-1456-4ba2-9c76-8f07f9296871" (UID: "c94dfe3c-1456-4ba2-9c76-8f07f9296871"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:37.258016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.257809    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache" pod "c94dfe3c-1456-4ba2-9c76-8f07f9296871" (UID: "c94dfe3c-1456-4ba2-9c76-8f07f9296871"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:37.258149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.258128    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:37.258212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.258153    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:37.258212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.258167    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:37.260656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.260624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm" pod "c94dfe3c-1456-4ba2-9c76-8f07f9296871" (UID: "c94dfe3c-1456-4ba2-9c76-8f07f9296871"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:37.260656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.260633    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs" pod "c94dfe3c-1456-4ba2-9c76-8f07f9296871" (UID: "c94dfe3c-1456-4ba2-9c76-8f07f9296871"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:37.260818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.260671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d" pod "c94dfe3c-1456-4ba2-9c76-8f07f9296871" (UID: "c94dfe3c-1456-4ba2-9c76-8f07f9296871"). InnerVolumeSpecName "kube-api-access-hp78d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:37.359285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.359246    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp78d\" (UniqueName: \"kubernetes.io/projected/c94dfe3c-1456-4ba2-9c76-8f07f9296871-kube-api-access-hp78d\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:37.359285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.359282    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c94dfe3c-1456-4ba2-9c76-8f07f9296871-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:37.359285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:37.359291    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c94dfe3c-1456-4ba2-9c76-8f07f9296871-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:38.159101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:38.159067    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"
Oct 30 12:33:38.219367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:38.219331    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"]
Oct 30 12:33:38.226107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:38.226074    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ae3538/test-model-ae3538-kserve-5b645f7d9-jlxbj"]
Oct 30 12:33:38.271096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:38.271064    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:38.338070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:38.338035    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" path="/var/lib/kubelet/pods/c94dfe3c-1456-4ba2-9c76-8f07f9296871/volumes"
Oct 30 12:33:40.065114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:40.065063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:41.791941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:41.791907    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:43.438243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:43.438208    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:45.153736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:45.153700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:45.897531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:45.897490    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"]
Oct 30 12:33:46.111767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.111732    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:46.196110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.196071    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"]
Oct 30 12:33:46.197250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.197227    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" containerName="main"
Oct 30 12:33:46.197383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.197252    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" containerName="main"
Oct 30 12:33:46.197383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.197294    5060 cpu_manager.go:414] "containerMap: removing container" podUID="732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" containerName="revoke-keys"
Oct 30 12:33:46.197383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.197303    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" containerName="revoke-keys"
Oct 30 12:33:46.197677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.197532    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94dfe3c-1456-4ba2-9c76-8f07f9296871" containerName="main"
Oct 30 12:33:46.197677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.197553    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" containerName="revoke-keys"
Oct 30 12:33:46.205993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.205964    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.221528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.221499    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:33:46.221705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.221541    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:33:46.233521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.230806    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:33:46.233521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.230877    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:33:46.236735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.235916    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"]
Oct 30 12:33:46.297593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.297550    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcm2\" (UniqueName: \"kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.297800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.297755    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.297889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.297869    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.399642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.399607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcm2\" (UniqueName: \"kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.399925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.399705    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.399925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.399803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.400634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.400607    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.404928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.404901    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.423796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.423763    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcm2\" (UniqueName: \"kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2\") pod \"maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.546543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.546436    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:46.865072 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:33:46.865042    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad14f1f3_e9d4_4e63_b638_089e0cbc1865.slice/crio-d45a26d488180a31f99112ff820111f84ef128b9a4e0bc60ce4c94cf1e7a877f WatchSource:0}: Error finding container d45a26d488180a31f99112ff820111f84ef128b9a4e0bc60ce4c94cf1e7a877f: Status 404 returned error can't find the container with id d45a26d488180a31f99112ff820111f84ef128b9a4e0bc60ce4c94cf1e7a877f
Oct 30 12:33:46.872260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:46.868344    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"]
Oct 30 12:33:47.035973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:47.035934    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:47.265054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:47.265019    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr" event={"ID":"ad14f1f3-e9d4-4e63-b638-089e0cbc1865","Type":"ContainerStarted","Data":"d45a26d488180a31f99112ff820111f84ef128b9a4e0bc60ce4c94cf1e7a877f"}
Oct 30 12:33:48.276166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:48.276062    5060 generic.go:356] "Generic (PLEG): container finished" podID="ad14f1f3-e9d4-4e63-b638-089e0cbc1865" containerID="4d5b9a3a2dc17c09c53d2077f1e933dd1ca154e979a188702a8e24443e9258a1" exitCode=0
Oct 30 12:33:48.276690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:48.276211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr" event={"ID":"ad14f1f3-e9d4-4e63-b638-089e0cbc1865","Type":"ContainerDied","Data":"4d5b9a3a2dc17c09c53d2077f1e933dd1ca154e979a188702a8e24443e9258a1"}
Oct 30 12:33:48.856375 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:48.856340    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:49.473359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.473333    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:49.548901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.548815    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2\" (UniqueName: \"kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2\") pod \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") "
Oct 30 12:33:49.548901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.548869    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca\") pod \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") "
Oct 30 12:33:49.549154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.548935    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca\") pod \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\" (UID: \"ad14f1f3-e9d4-4e63-b638-089e0cbc1865\") "
Oct 30 12:33:49.549348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.549312    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca" pod "ad14f1f3-e9d4-4e63-b638-089e0cbc1865" (UID: "ad14f1f3-e9d4-4e63-b638-089e0cbc1865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:49.549717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.549692    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:49.551227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.551194    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2" pod "ad14f1f3-e9d4-4e63-b638-089e0cbc1865" (UID: "ad14f1f3-e9d4-4e63-b638-089e0cbc1865"). InnerVolumeSpecName "kube-api-access-mjcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:49.551478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.551425    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca" pod "ad14f1f3-e9d4-4e63-b638-089e0cbc1865" (UID: "ad14f1f3-e9d4-4e63-b638-089e0cbc1865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:49.650809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.650772    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcm2\" (UniqueName: \"kubernetes.io/projected/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-kube-api-access-mjcm2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:49.650809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:49.650802    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad14f1f3-e9d4-4e63-b638-089e0cbc1865-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:33:49.858656 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:33:49.858571    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:33:50.300324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:50.300294    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"
Oct 30 12:33:50.300560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:50.300326    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr" event={"ID":"ad14f1f3-e9d4-4e63-b638-089e0cbc1865","Type":"ContainerDied","Data":"d45a26d488180a31f99112ff820111f84ef128b9a4e0bc60ce4c94cf1e7a877f"}
Oct 30 12:33:50.300560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:50.300379    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45a26d488180a31f99112ff820111f84ef128b9a4e0bc60ce4c94cf1e7a877f"
Oct 30 12:33:50.501521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:50.501411    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:52.251139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:52.251085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:54.042210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:54.042174    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:55.616606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:55.616569    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:55.831116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:55.831075    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:56.819692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:56.819656    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:56.968679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:56.968639    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:57.680634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:57.680601    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:33:59.442969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:33:59.442890    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:00.942924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:00.942861    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g" podUID="e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" containerName="main" containerID="cri-o://87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e" gracePeriod=45
Oct 30 12:34:01.207124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.207043    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:01.280006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.279973    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:34:01.299760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.299721    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home\") pod \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") "
Oct 30 12:34:01.299982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.299776    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache\") pod \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") "
Oct 30 12:34:01.299982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.299831    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs\") pod \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") "
Oct 30 12:34:01.299982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.299887    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm\") pod \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") "
Oct 30 12:34:01.299982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.299914    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir\") pod \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") "
Oct 30 12:34:01.300242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.300042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home" pod "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" (UID: "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:01.300242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.300084    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache" pod "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" (UID: "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:01.300242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.300149    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5\" (UniqueName: \"kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5\") pod \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\" (UID: \"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b\") "
Oct 30 12:34:01.300408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.300288    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir" pod "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" (UID: "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:01.301107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.301081    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:01.301107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.301112    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:01.301324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.301153    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:01.302748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.302720    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm" pod "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" (UID: "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:01.303032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.302998    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs" pod "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" (UID: "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:34:01.303216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.303194    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5" pod "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" (UID: "e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b"). InnerVolumeSpecName "kube-api-access-mk7f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:01.403222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.403181    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:01.403222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.403209    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:01.403222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.403222    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7f5\" (UniqueName: \"kubernetes.io/projected/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b-kube-api-access-mk7f5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:01.490231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.490089    5060 generic.go:356] "Generic (PLEG): container finished" podID="e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" containerID="87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e" exitCode=0
Oct 30 12:34:01.490231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.490136    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"
Oct 30 12:34:01.490231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.490177    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g" event={"ID":"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b","Type":"ContainerDied","Data":"87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e"}
Oct 30 12:34:01.490231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.490223    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g" event={"ID":"e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b","Type":"ContainerDied","Data":"aedf6d36e10897b258d221d343d51248937822966fa7a1077dc972647293d119"}
Oct 30 12:34:01.490667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.490248    5060 scope.go:122] "RemoveContainer" containerID="87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e"
Oct 30 12:34:01.504439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.504415    5060 scope.go:122] "RemoveContainer" containerID="87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e"
Oct 30 12:34:01.504832 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:34:01.504797    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e\": container with ID starting with 87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e not found: ID does not exist" containerID="87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e"
Oct 30 12:34:01.504905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.504846    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e"} err="failed to get container status \"87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e\": rpc error: code = NotFound desc = could not find container \"87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e\": container with ID starting with 87034cbe888761087a9170d7a71a4a876928b5ee7ea75db25502089db17f195e not found: ID does not exist"
Oct 30 12:34:01.546162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.546126    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"]
Oct 30 12:34:01.554465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:01.554419    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0ecae0/test-model-0ecae0-kserve-689f596687-wmf4g"]
Oct 30 12:34:02.337662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:02.337630    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" path="/var/lib/kubelet/pods/e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b/volumes"
Oct 30 12:34:02.934853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:02.934809    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:03.003521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:03.003485    5060 scope.go:122] "RemoveContainer" containerID="1bf1ab6f5b875d63f5e15a31cef9610a7fa63a75670b54133f396d64b248ef4e"
Oct 30 12:34:04.755329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:04.755296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:06.616391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:06.616341    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:08.314531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:08.314475    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:10.049652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:10.049617    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:11.839241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:11.839207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:12.713906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:12.713867    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:12.865182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:12.865146    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:13.315146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:13.315108    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:13.898268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:13.898206    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:14.061299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:14.061255    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:14.073371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:14.073326    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"]
Oct 30 12:34:14.087125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:14.087085    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae3538-xcmmr"]
Oct 30 12:34:14.205352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:14.205319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:14.336932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:14.336898    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad14f1f3-e9d4-4e63-b638-089e0cbc1865" path="/var/lib/kubelet/pods/ad14f1f3-e9d4-4e63-b638-089e0cbc1865/volumes"
Oct 30 12:34:14.596287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:14.596189    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:15.875702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:15.875665    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:17.617034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:17.616987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:19.435523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:19.435447    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:19.859498 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:34:19.859440    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:34:20.906664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:20.906610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:21.060558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.060502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:21.189196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.189105    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"]
Oct 30 12:34:21.189745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.189689    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy" containerID="cri-o://ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56" gracePeriod=30
Oct 30 12:34:21.267582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.266266    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:21.432054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.432013    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:21.828399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.828362    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"]
Oct 30 12:34:21.829503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.829479    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ad14f1f3-e9d4-4e63-b638-089e0cbc1865" containerName="revoke-keys"
Oct 30 12:34:21.829613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.829505    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad14f1f3-e9d4-4e63-b638-089e0cbc1865" containerName="revoke-keys"
Oct 30 12:34:21.829613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.829567    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" containerName="main"
Oct 30 12:34:21.829613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.829574    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" containerName="main"
Oct 30 12:34:21.829787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.829773    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad14f1f3-e9d4-4e63-b638-089e0cbc1865" containerName="revoke-keys"
Oct 30 12:34:21.829826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.829801    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7eaa66b-7e75-4bb9-9fd1-f21e7f2e4d4b" containerName="main"
Oct 30 12:34:21.836035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.836007    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:21.851120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.851088    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:34:21.851120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.851089    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"]
Oct 30 12:34:21.851346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.851290    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:34:21.863577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.863551    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:34:21.863787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.863768    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:34:21.944150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.944114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:21.944150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.944152    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jng8\" (UniqueName: \"kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:21.944623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:21.944232    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.045743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.045698    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.045945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.045749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jng8\" (UniqueName: \"kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.045945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.045793    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.046509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.046445    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.048684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.048620    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.069499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.069468    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jng8\" (UniqueName: \"kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8\") pod \"maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.156570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.156493    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:22.469184 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:34:22.469154    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac07f73_0c76_4e87_bdaf_e5e01cf20256.slice/crio-3ce7f3b4849d4c012631faee76e43b053da6329e771ca599dea12b0451e8b51a WatchSource:0}: Error finding container 3ce7f3b4849d4c012631faee76e43b053da6329e771ca599dea12b0451e8b51a: Status 404 returned error can't find the container with id 3ce7f3b4849d4c012631faee76e43b053da6329e771ca599dea12b0451e8b51a
Oct 30 12:34:22.472914 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.472887    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"]
Oct 30 12:34:22.717175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:22.717083    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs" event={"ID":"aac07f73-0c76-4e87-bdaf-e5e01cf20256","Type":"ContainerStarted","Data":"3ce7f3b4849d4c012631faee76e43b053da6329e771ca599dea12b0451e8b51a"}
Oct 30 12:34:23.194932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:23.194896    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:23.557890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:23.557853    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:23.728528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:23.728491    5060 generic.go:356] "Generic (PLEG): container finished" podID="aac07f73-0c76-4e87-bdaf-e5e01cf20256" containerID="73c7b2deee14689ed5365990240db85cad69b5b18991b7b2c72ab4851e156e0e" exitCode=0
Oct 30 12:34:23.728803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:23.728575    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs" event={"ID":"aac07f73-0c76-4e87-bdaf-e5e01cf20256","Type":"ContainerDied","Data":"73c7b2deee14689ed5365990240db85cad69b5b18991b7b2c72ab4851e156e0e"}
Oct 30 12:34:24.633745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.633709    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:24.926732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.926707    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:25.002508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.990850    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca\") pod \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") "
Oct 30 12:34:25.002508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.991071    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca\") pod \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") "
Oct 30 12:34:25.002508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.991252    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8\" (UniqueName: \"kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8\") pod \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\" (UID: \"aac07f73-0c76-4e87-bdaf-e5e01cf20256\") "
Oct 30 12:34:25.002508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.991267    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca" pod "aac07f73-0c76-4e87-bdaf-e5e01cf20256" (UID: "aac07f73-0c76-4e87-bdaf-e5e01cf20256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:25.002508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:24.992339    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:25.003433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.003399    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8" pod "aac07f73-0c76-4e87-bdaf-e5e01cf20256" (UID: "aac07f73-0c76-4e87-bdaf-e5e01cf20256"). InnerVolumeSpecName "kube-api-access-5jng8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:25.003615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.003563    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca" pod "aac07f73-0c76-4e87-bdaf-e5e01cf20256" (UID: "aac07f73-0c76-4e87-bdaf-e5e01cf20256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:34:25.041678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.041638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:25.094403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.094356    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac07f73-0c76-4e87-bdaf-e5e01cf20256-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:25.094403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.094400    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jng8\" (UniqueName: \"kubernetes.io/projected/aac07f73-0c76-4e87-bdaf-e5e01cf20256-kube-api-access-5jng8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:25.749689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.749649    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"
Oct 30 12:34:25.749689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.749660    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs" event={"ID":"aac07f73-0c76-4e87-bdaf-e5e01cf20256","Type":"ContainerDied","Data":"3ce7f3b4849d4c012631faee76e43b053da6329e771ca599dea12b0451e8b51a"}
Oct 30 12:34:25.750187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:25.749712    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce7f3b4849d4c012631faee76e43b053da6329e771ca599dea12b0451e8b51a"
Oct 30 12:34:26.496873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.496841    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:34:26.617087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617034    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617124    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617157    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617202    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617240    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617298    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617485    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617502    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.617714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617636    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.617714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.617675    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo\") pod \"33e7b99b-42cc-48c7-806b-ed940feee094\" (UID: \"33e7b99b-42cc-48c7-806b-ed940feee094\") "
Oct 30 12:34:26.618473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.618434    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.618687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.618559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.618821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.618580    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.619198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.619159    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:26.619578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.619538    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.620595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.620559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.620708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.620636    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "kube-api-access-mr52s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:26.620708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.620671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:26.621539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.621515    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo" pod "33e7b99b-42cc-48c7-806b-ed940feee094" (UID: "33e7b99b-42cc-48c7-806b-ed940feee094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:34:26.720056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720019    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720057    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720069    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720078    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr52s\" (UniqueName: \"kubernetes.io/projected/33e7b99b-42cc-48c7-806b-ed940feee094-kube-api-access-mr52s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720088    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720097    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e7b99b-42cc-48c7-806b-ed940feee094-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720106    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e7b99b-42cc-48c7-806b-ed940feee094-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.720302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.720114    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e7b99b-42cc-48c7-806b-ed940feee094-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.760476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.760372    5060 generic.go:356] "Generic (PLEG): container finished" podID="33e7b99b-42cc-48c7-806b-ed940feee094" containerID="ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56" exitCode=0
Oct 30 12:34:26.760946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.760480    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"
Oct 30 12:34:26.760946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.760489    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" event={"ID":"33e7b99b-42cc-48c7-806b-ed940feee094","Type":"ContainerDied","Data":"ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56"}
Oct 30 12:34:26.760946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.760545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864" event={"ID":"33e7b99b-42cc-48c7-806b-ed940feee094","Type":"ContainerDied","Data":"421d47d3c30f8af86bdc7e5424cb829d6c97950cc14a1a82034161de90cb2f52"}
Oct 30 12:34:26.760946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.760571    5060 scope.go:122] "RemoveContainer" containerID="ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56"
Oct 30 12:34:26.773266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.773248    5060 scope.go:122] "RemoveContainer" containerID="d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"
Oct 30 12:34:26.783818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.783794    5060 scope.go:122] "RemoveContainer" containerID="ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56"
Oct 30 12:34:26.784111 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:34:26.784086    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56\": container with ID starting with ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56 not found: ID does not exist" containerID="ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56"
Oct 30 12:34:26.784198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.784123    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56"} err="failed to get container status \"ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56\": rpc error: code = NotFound desc = could not find container \"ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56\": container with ID starting with ec5e9ace764e8b5b4d775378e118a763be3e1ad3e2cd31f7cdb7943fe8c4cc56 not found: ID does not exist"
Oct 30 12:34:26.784198 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.784149    5060 scope.go:122] "RemoveContainer" containerID="d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"
Oct 30 12:34:26.784501 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:34:26.784475    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f\": container with ID starting with d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f not found: ID does not exist" containerID="d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"
Oct 30 12:34:26.784616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.784512    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f"} err="failed to get container status \"d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f\": rpc error: code = NotFound desc = could not find container \"d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f\": container with ID starting with d1fa3a9a491b1af75ff113abc2f43724d60a75d88f877a98d6547f1ba21b7b7f not found: ID does not exist"
Oct 30 12:34:26.792076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.792047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:26.831480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.827987    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"]
Oct 30 12:34:26.841439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:26.839683    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-sw864"]
Oct 30 12:34:28.339716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:28.339684    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" path="/var/lib/kubelet/pods/33e7b99b-42cc-48c7-806b-ed940feee094/volumes"
Oct 30 12:34:28.654932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:28.654838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:30.386367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:30.386328    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:30.687057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:30.686967    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:31.032893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:31.032856    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"]
Oct 30 12:34:31.045064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:31.045033    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f4b1a3-bjwrn"]
Oct 30 12:34:32.075948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:32.075915    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:32.259628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:32.259588    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:32.337775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:32.337686    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6581ae-7654-4848-82e3-dbac5e3ccec8" path="/var/lib/kubelet/pods/cc6581ae-7654-4848-82e3-dbac5e3ccec8/volumes"
Oct 30 12:34:32.338096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:32.338080    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"]
Oct 30 12:34:32.392720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:32.392623    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:32.575814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:32.575778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:33.905004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:33.904958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:35.546267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:35.546228    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:37.261113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:37.261064    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:37.338871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:37.338833    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" containerName="payload-pre-processing" containerID="cri-o://44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b" gracePeriod=25
Oct 30 12:34:39.009857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:39.009821    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:39.679100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:39.679061    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:39.824763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:39.824722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:40.763464 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:40.763425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:42.454088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:42.454050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:43.031935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.031895    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"]
Oct 30 12:34:43.043989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.043959    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afad348-vj4cz"]
Oct 30 12:34:43.530555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.530514    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"]
Oct 30 12:34:43.531422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531398    5060 cpu_manager.go:414] "containerMap: removing container" podUID="aac07f73-0c76-4e87-bdaf-e5e01cf20256" containerName="revoke-keys"
Oct 30 12:34:43.531568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531423    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac07f73-0c76-4e87-bdaf-e5e01cf20256" containerName="revoke-keys"
Oct 30 12:34:43.531568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531505    5060 cpu_manager.go:414] "containerMap: removing container" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy"
Oct 30 12:34:43.531568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531514    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy"
Oct 30 12:34:43.531568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531534    5060 cpu_manager.go:414] "containerMap: removing container" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy"
Oct 30 12:34:43.531568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531542    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy"
Oct 30 12:34:43.531857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531759    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac07f73-0c76-4e87-bdaf-e5e01cf20256" containerName="revoke-keys"
Oct 30 12:34:43.531857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531785    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy"
Oct 30 12:34:43.531857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.531798    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e7b99b-42cc-48c7-806b-ed940feee094" containerName="istio-proxy"
Oct 30 12:34:43.535741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.535714    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.550997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.550961    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:34:43.550997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.550984    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:34:43.551190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.550976    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:34:43.551190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.550969    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:34:43.551268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.551245    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"]
Oct 30 12:34:43.603734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.603698    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prs78\" (UniqueName: \"kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.603943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.603822    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.604048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.604019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.705968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.705937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.706182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.706012    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prs78\" (UniqueName: \"kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.706182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.706083    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.706722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.706695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.708794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.708771    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.729370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.729347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prs78\" (UniqueName: \"kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78\") pod \"maas-api-revoke-sub-c21b2303-ht656\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:43.851875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:43.851799    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:44.122545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:44.122427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:44.180698 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:34:44.180662    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997c97d3_902a_4f6c_876c_a69814e8beeb.slice/crio-615e4bcf7befde95243c89a4930c707a8347c5de60a623ed6e791b05ca7c218e WatchSource:0}: Error finding container 615e4bcf7befde95243c89a4930c707a8347c5de60a623ed6e791b05ca7c218e: Status 404 returned error can't find the container with id 615e4bcf7befde95243c89a4930c707a8347c5de60a623ed6e791b05ca7c218e
Oct 30 12:34:44.184873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:44.184845    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"]
Oct 30 12:34:44.343094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:44.343061    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1" path="/var/lib/kubelet/pods/f0a860a9-f5c9-4c22-b9d9-d90cc17c4be1/volumes"
Oct 30 12:34:44.945639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:44.945596    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656" event={"ID":"997c97d3-902a-4f6c-876c-a69814e8beeb","Type":"ContainerStarted","Data":"ae80d49d4a862e4533e1a261c3bda494aa64a58b962a49b12a6930c46470afa4"}
Oct 30 12:34:44.946056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:44.945659    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656" event={"ID":"997c97d3-902a-4f6c-876c-a69814e8beeb","Type":"ContainerStarted","Data":"615e4bcf7befde95243c89a4930c707a8347c5de60a623ed6e791b05ca7c218e"}
Oct 30 12:34:44.986839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:44.986775    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656" podStartSLOduration=1.9867573379999999 podStartE2EDuration="1.986757338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:44.985984178 +0000 UTC m=+3888.780868674" watchObservedRunningTime="2026-10-30 12:34:44.986757338 +0000 UTC m=+3888.781641834"
Oct 30 12:34:45.860525 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:45.860487    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:45.960127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:45.960092    5060 generic.go:356] "Generic (PLEG): container finished" podID="997c97d3-902a-4f6c-876c-a69814e8beeb" containerID="ae80d49d4a862e4533e1a261c3bda494aa64a58b962a49b12a6930c46470afa4" exitCode=0
Oct 30 12:34:45.960611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:45.960242    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656" event={"ID":"997c97d3-902a-4f6c-876c-a69814e8beeb","Type":"ContainerDied","Data":"ae80d49d4a862e4533e1a261c3bda494aa64a58b962a49b12a6930c46470afa4"}
Oct 30 12:34:46.296078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:46.296033    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:34:47.152124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.152095    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:47.261253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.261165    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca\") pod \"997c97d3-902a-4f6c-876c-a69814e8beeb\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") "
Oct 30 12:34:47.261425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.261291    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca\") pod \"997c97d3-902a-4f6c-876c-a69814e8beeb\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") "
Oct 30 12:34:47.261536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.261514    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78\" (UniqueName: \"kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78\") pod \"997c97d3-902a-4f6c-876c-a69814e8beeb\" (UID: \"997c97d3-902a-4f6c-876c-a69814e8beeb\") "
Oct 30 12:34:47.261596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.261560    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca" pod "997c97d3-902a-4f6c-876c-a69814e8beeb" (UID: "997c97d3-902a-4f6c-876c-a69814e8beeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:47.262191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.262167    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:47.263633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.263611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca" pod "997c97d3-902a-4f6c-876c-a69814e8beeb" (UID: "997c97d3-902a-4f6c-876c-a69814e8beeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:34:47.263880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.263859    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78" pod "997c97d3-902a-4f6c-876c-a69814e8beeb" (UID: "997c97d3-902a-4f6c-876c-a69814e8beeb"). InnerVolumeSpecName "kube-api-access-prs78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:47.363943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.363906    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prs78\" (UniqueName: \"kubernetes.io/projected/997c97d3-902a-4f6c-876c-a69814e8beeb-kube-api-access-prs78\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:47.363943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.363940    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997c97d3-902a-4f6c-876c-a69814e8beeb-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:47.653228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.653136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:47.979445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.979401    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656" event={"ID":"997c97d3-902a-4f6c-876c-a69814e8beeb","Type":"ContainerDied","Data":"615e4bcf7befde95243c89a4930c707a8347c5de60a623ed6e791b05ca7c218e"}
Oct 30 12:34:47.979445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.979434    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"
Oct 30 12:34:47.979758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:47.979487    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="615e4bcf7befde95243c89a4930c707a8347c5de60a623ed6e791b05ca7c218e"
Oct 30 12:34:49.382347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:49.382301    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:49.858259 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:34:49.858225    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:34:51.148757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:51.148718    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:52.806891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:52.806857    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:52.963591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:52.963555    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:54.433076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:54.433040    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"]
Oct 30 12:34:54.501999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:54.501963    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:54.672476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:54.672050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:55.030579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.030547    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"]
Oct 30 12:34:55.031299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.031284    5060 cpu_manager.go:414] "containerMap: removing container" podUID="997c97d3-902a-4f6c-876c-a69814e8beeb" containerName="revoke-keys"
Oct 30 12:34:55.031367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.031300    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c97d3-902a-4f6c-876c-a69814e8beeb" containerName="revoke-keys"
Oct 30 12:34:55.031427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.031418    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="997c97d3-902a-4f6c-876c-a69814e8beeb" containerName="revoke-keys"
Oct 30 12:34:55.035445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.035426    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.051632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.051601    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"]
Oct 30 12:34:55.058672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.058642    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:34:55.058815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.058728    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:34:55.116780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.116749    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.116996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.116864    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.116996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.116905    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqrg\" (UniqueName: \"kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.124995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.124966    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:34:55.124995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.124990    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:34:55.217997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.217956    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.218206 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.218017    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqrg\" (UniqueName: \"kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.218336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.218314    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.219130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.219107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.220721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.220699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.241732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.241709    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqrg\" (UniqueName: \"kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg\") pod \"maas-api-revoke-sub-06d541f9-cb8cw\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.354056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.353966    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:55.673692 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:34:55.673659    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c02e1df_4394_4588_b6c0_c417fd4f2bd9.slice/crio-4100a8c2e6e3bd8a7451726b30bf2ac785d43d0a6c82d4d1c7347fcb37ea7912 WatchSource:0}: Error finding container 4100a8c2e6e3bd8a7451726b30bf2ac785d43d0a6c82d4d1c7347fcb37ea7912: Status 404 returned error can't find the container with id 4100a8c2e6e3bd8a7451726b30bf2ac785d43d0a6c82d4d1c7347fcb37ea7912
Oct 30 12:34:55.677209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.677182    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"]
Oct 30 12:34:55.741678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:55.741648    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:56.060607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:56.060562    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw" event={"ID":"9c02e1df-4394-4588-b6c0-c417fd4f2bd9","Type":"ContainerStarted","Data":"4100a8c2e6e3bd8a7451726b30bf2ac785d43d0a6c82d4d1c7347fcb37ea7912"}
Oct 30 12:34:56.061528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:56.061501    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:56.296013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:56.295976    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:34:56.562743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:56.561137    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:57.072231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.072196    5060 generic.go:356] "Generic (PLEG): container finished" podID="9c02e1df-4394-4588-b6c0-c417fd4f2bd9" containerID="5f82ddbe09bad9040d8ec33f6234a8bb1ac8f87aaffc57eb425a36bdc5ae564e" exitCode=0
Oct 30 12:34:57.072779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.072290    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw" event={"ID":"9c02e1df-4394-4588-b6c0-c417fd4f2bd9","Type":"ContainerDied","Data":"5f82ddbe09bad9040d8ec33f6234a8bb1ac8f87aaffc57eb425a36bdc5ae564e"}
Oct 30 12:34:57.660379 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.660345    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:34:57.781679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.781649    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl\" (UniqueName: \"kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl\") pod \"d0721f4c-5439-493b-8383-73ef1e0051db\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") "
Oct 30 12:34:57.781895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.781825    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume\") pod \"d0721f4c-5439-493b-8383-73ef1e0051db\" (UID: \"d0721f4c-5439-493b-8383-73ef1e0051db\") "
Oct 30 12:34:57.782238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.782209    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume" pod "d0721f4c-5439-493b-8383-73ef1e0051db" (UID: "d0721f4c-5439-493b-8383-73ef1e0051db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:57.782437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.782417    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0721f4c-5439-493b-8383-73ef1e0051db-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:57.787495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.784243    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl" pod "d0721f4c-5439-493b-8383-73ef1e0051db" (UID: "d0721f4c-5439-493b-8383-73ef1e0051db"). InnerVolumeSpecName "kube-api-access-9z2vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:57.825164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.825131    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:34:57.830374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.830351    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:34:57.884697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:57.884567    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2vl\" (UniqueName: \"kubernetes.io/projected/d0721f4c-5439-493b-8383-73ef1e0051db-kube-api-access-9z2vl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:58.083116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.083075    5060 generic.go:356] "Generic (PLEG): container finished" podID="d0721f4c-5439-493b-8383-73ef1e0051db" containerID="44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b" exitCode=0
Oct 30 12:34:58.083627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.083117    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"
Oct 30 12:34:58.083627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.083155    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" event={"ID":"d0721f4c-5439-493b-8383-73ef1e0051db","Type":"ContainerDied","Data":"44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b"}
Oct 30 12:34:58.083627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.083235    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g" event={"ID":"d0721f4c-5439-493b-8383-73ef1e0051db","Type":"ContainerDied","Data":"66935490bfe112935ab618eb1e9809f0ee1d1f2ef60aaf0cc7b366fb25b63070"}
Oct 30 12:34:58.083627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.083256    5060 scope.go:122] "RemoveContainer" containerID="44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b"
Oct 30 12:34:58.098852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.098830    5060 scope.go:122] "RemoveContainer" containerID="44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b"
Oct 30 12:34:58.099246 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:34:58.099215    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b\": container with ID starting with 44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b not found: ID does not exist" containerID="44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b"
Oct 30 12:34:58.099350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.099256    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b"} err="failed to get container status \"44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b\": rpc error: code = NotFound desc = could not find container \"44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b\": container with ID starting with 44c9f5f8a846e0c21d37c14b810e496474f5d87620b8d1e4a981a644fd392e6b not found: ID does not exist"
Oct 30 12:34:58.140374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.140280    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"]
Oct 30 12:34:58.158338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.151595    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ecae0-7657485d7-4p99g"]
Oct 30 12:34:58.277333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.277301    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:58.339969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.339935    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" path="/var/lib/kubelet/pods/d0721f4c-5439-493b-8383-73ef1e0051db/volumes"
Oct 30 12:34:58.373202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.373167    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:34:58.413957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.412817    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg\" (UniqueName: \"kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg\") pod \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") "
Oct 30 12:34:58.413957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.412945    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca\") pod \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") "
Oct 30 12:34:58.413957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.412983    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca\") pod \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\" (UID: \"9c02e1df-4394-4588-b6c0-c417fd4f2bd9\") "
Oct 30 12:34:58.420485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.419317    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca" pod "9c02e1df-4394-4588-b6c0-c417fd4f2bd9" (UID: "9c02e1df-4394-4588-b6c0-c417fd4f2bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:58.420485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.419940    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:58.435929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.421664    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca" pod "9c02e1df-4394-4588-b6c0-c417fd4f2bd9" (UID: "9c02e1df-4394-4588-b6c0-c417fd4f2bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:34:58.435929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.423694    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg" pod "9c02e1df-4394-4588-b6c0-c417fd4f2bd9" (UID: "9c02e1df-4394-4588-b6c0-c417fd4f2bd9"). InnerVolumeSpecName "kube-api-access-lzqrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:58.521777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.521741    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqrg\" (UniqueName: \"kubernetes.io/projected/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-kube-api-access-lzqrg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:58.521777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:58.521779    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02e1df-4394-4588-b6c0-c417fd4f2bd9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:34:59.092677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:59.092643    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"
Oct 30 12:34:59.093127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:59.092638    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw" event={"ID":"9c02e1df-4394-4588-b6c0-c417fd4f2bd9","Type":"ContainerDied","Data":"4100a8c2e6e3bd8a7451726b30bf2ac785d43d0a6c82d4d1c7347fcb37ea7912"}
Oct 30 12:34:59.093127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:34:59.092787    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4100a8c2e6e3bd8a7451726b30bf2ac785d43d0a6c82d4d1c7347fcb37ea7912"
Oct 30 12:35:00.044277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:00.044241    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:01.706882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:01.706835    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:02.636162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:02.636120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:03.465216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:03.465174    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:05.164874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:05.162197    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:07.047160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:07.047052    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:08.870134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:08.870090    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:09.461941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.461896    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr" podUID="cf525b07-aa77-49bb-aec4-dafb8d5b3a27" containerName="main" containerID="cri-o://880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e" gracePeriod=45
Oct 30 12:35:09.757825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.757797    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:35:09.808920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.808886    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home\") pod \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") "
Oct 30 12:35:09.809121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.808943    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs\") pod \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") "
Oct 30 12:35:09.809121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.809079    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch\" (UniqueName: \"kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch\") pod \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") "
Oct 30 12:35:09.809260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.809205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm\") pod \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") "
Oct 30 12:35:09.809260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.809254    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir\") pod \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") "
Oct 30 12:35:09.809382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.809310    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache\") pod \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\" (UID: \"cf525b07-aa77-49bb-aec4-dafb8d5b3a27\") "
Oct 30 12:35:09.809549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.809521    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home" pod "cf525b07-aa77-49bb-aec4-dafb8d5b3a27" (UID: "cf525b07-aa77-49bb-aec4-dafb8d5b3a27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:09.810009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.809990    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:09.810563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.810540    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache" pod "cf525b07-aa77-49bb-aec4-dafb8d5b3a27" (UID: "cf525b07-aa77-49bb-aec4-dafb8d5b3a27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:09.811294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.811240    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir" pod "cf525b07-aa77-49bb-aec4-dafb8d5b3a27" (UID: "cf525b07-aa77-49bb-aec4-dafb8d5b3a27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:09.813708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.813680    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch" pod "cf525b07-aa77-49bb-aec4-dafb8d5b3a27" (UID: "cf525b07-aa77-49bb-aec4-dafb8d5b3a27"). InnerVolumeSpecName "kube-api-access-xxsch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:09.814361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.814333    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs" pod "cf525b07-aa77-49bb-aec4-dafb8d5b3a27" (UID: "cf525b07-aa77-49bb-aec4-dafb8d5b3a27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:09.821336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.821301    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm" pod "cf525b07-aa77-49bb-aec4-dafb8d5b3a27" (UID: "cf525b07-aa77-49bb-aec4-dafb8d5b3a27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:09.911278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.911243    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxsch\" (UniqueName: \"kubernetes.io/projected/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-kube-api-access-xxsch\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:09.911278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.911274    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:09.911812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.911289    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:09.911812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.911303    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:09.911812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:09.911315    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf525b07-aa77-49bb-aec4-dafb8d5b3a27-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:10.201793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.201756    5060 generic.go:356] "Generic (PLEG): container finished" podID="cf525b07-aa77-49bb-aec4-dafb8d5b3a27" containerID="880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e" exitCode=0
Oct 30 12:35:10.202015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.201813    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"
Oct 30 12:35:10.202015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.201847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr" event={"ID":"cf525b07-aa77-49bb-aec4-dafb8d5b3a27","Type":"ContainerDied","Data":"880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e"}
Oct 30 12:35:10.202015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.201905    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr" event={"ID":"cf525b07-aa77-49bb-aec4-dafb8d5b3a27","Type":"ContainerDied","Data":"4fad2271a70861f6b0e2604210626a82d82e7c20b082d9e0cd18526cc1fc7723"}
Oct 30 12:35:10.202015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.201929    5060 scope.go:122] "RemoveContainer" containerID="880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e"
Oct 30 12:35:10.223048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.223002    5060 scope.go:122] "RemoveContainer" containerID="880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e"
Oct 30 12:35:10.257617 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:35:10.257571    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e\": container with ID starting with 880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e not found: ID does not exist" containerID="880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e"
Oct 30 12:35:10.257836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.257762    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e"} err="failed to get container status \"880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e\": rpc error: code = NotFound desc = could not find container \"880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e\": container with ID starting with 880932f9c3d4ea270f8ba1cf39a70b64e5335c83b992d720c8716ec9882d4a1e not found: ID does not exist"
Oct 30 12:35:10.274473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.274418    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"]
Oct 30 12:35:10.282284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.282243    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0190d8/test-model-0190d8-kserve-c5d874c47-zbfpr"]
Oct 30 12:35:10.335404 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.335338    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:10.337731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.337704    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf525b07-aa77-49bb-aec4-dafb8d5b3a27" path="/var/lib/kubelet/pods/cf525b07-aa77-49bb-aec4-dafb8d5b3a27/volumes"
Oct 30 12:35:10.632537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.625332    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:10.879684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.879631    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:10.997775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:10.997731    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"]
Oct 30 12:35:11.012320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:11.012286    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ecae0-4wcbs"]
Oct 30 12:35:11.035168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:11.035129    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:11.176358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:11.176309    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:11.525539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:11.525447    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:12.336252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:12.336212    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac07f73-0c76-4e87-bdaf-e5e01cf20256" path="/var/lib/kubelet/pods/aac07f73-0c76-4e87-bdaf-e5e01cf20256/volumes"
Oct 30 12:35:12.815271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:12.815235    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:14.670191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:14.670153    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:16.346656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:16.346618    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:17.594474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.594400    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"]
Oct 30 12:35:17.662538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.662502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:17.960013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.959961    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"]
Oct 30 12:35:17.961336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.961305    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" containerName="payload-pre-processing"
Oct 30 12:35:17.961542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.961524    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" containerName="payload-pre-processing"
Oct 30 12:35:17.961680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.961670    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cf525b07-aa77-49bb-aec4-dafb8d5b3a27" containerName="main"
Oct 30 12:35:17.961758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.961749    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf525b07-aa77-49bb-aec4-dafb8d5b3a27" containerName="main"
Oct 30 12:35:17.961893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.961882    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9c02e1df-4394-4588-b6c0-c417fd4f2bd9" containerName="revoke-keys"
Oct 30 12:35:17.961986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.961974    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c02e1df-4394-4588-b6c0-c417fd4f2bd9" containerName="revoke-keys"
Oct 30 12:35:17.962284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.962266    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf525b07-aa77-49bb-aec4-dafb8d5b3a27" containerName="main"
Oct 30 12:35:17.962471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.962439    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c02e1df-4394-4588-b6c0-c417fd4f2bd9" containerName="revoke-keys"
Oct 30 12:35:17.962618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.962604    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0721f4c-5439-493b-8383-73ef1e0051db" containerName="payload-pre-processing"
Oct 30 12:35:17.970593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.969209    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:17.986582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.986551    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:35:17.990545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.990518    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"]
Oct 30 12:35:17.996108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:17.996087    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:35:18.017566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.017406    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:35:18.017928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.017910    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:35:18.059156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.059112    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.059369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.059338    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.059493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.059409    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qpq\" (UniqueName: \"kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.161135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.161099    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qpq\" (UniqueName: \"kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.161316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.161260    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.161495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.161472    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.162114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.162088    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.164103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.164073    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.186334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.186309    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qpq\" (UniqueName: \"kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq\") pod \"maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.194591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.194562    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:18.304162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.304082    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:18.581538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.575329    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:18.701501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.693209    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"]
Oct 30 12:35:18.722472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.722418    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:18.945693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:18.945667    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:19.296962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:19.296924    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99" event={"ID":"7bb5f12d-84d2-4a52-8802-d9e0381f4c90","Type":"ContainerStarted","Data":"55fb46b87eae2c6fbc74b389cb8efe8d4d3697f94c6aa7f91f83fec3718a7eda"}
Oct 30 12:35:19.297174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:19.296981    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99" event={"ID":"7bb5f12d-84d2-4a52-8802-d9e0381f4c90","Type":"ContainerStarted","Data":"12d221e811a7d257a094d3ec7823ce5cf5145a036d91082289dfdaa05405b22e"}
Oct 30 12:35:19.344537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:19.344445    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99" podStartSLOduration=2.344426127 podStartE2EDuration="2.344426127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:19.337348106 +0000 UTC m=+3923.132232600" watchObservedRunningTime="2026-10-30 12:35:19.344426127 +0000 UTC m=+3923.139310620"
Oct 30 12:35:19.860825 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:35:19.860781    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:35:19.960601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:19.960567    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:20.109040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:20.108929    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:20.255311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:20.255208    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:20.316759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:20.316725    5060 generic.go:356] "Generic (PLEG): container finished" podID="7bb5f12d-84d2-4a52-8802-d9e0381f4c90" containerID="55fb46b87eae2c6fbc74b389cb8efe8d4d3697f94c6aa7f91f83fec3718a7eda" exitCode=0
Oct 30 12:35:20.316968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:20.316809    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99" event={"ID":"7bb5f12d-84d2-4a52-8802-d9e0381f4c90","Type":"ContainerDied","Data":"55fb46b87eae2c6fbc74b389cb8efe8d4d3697f94c6aa7f91f83fec3718a7eda"}
Oct 30 12:35:20.393536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:20.393390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:21.172273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.172233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:21.315471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.315418    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:21.488465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.488428    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:21.499287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.499263    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:21.561955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.561917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq\" (UniqueName: \"kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq\") pod \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") "
Oct 30 12:35:21.562284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.562258    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca\") pod \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") "
Oct 30 12:35:21.562367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.562336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca\") pod \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\" (UID: \"7bb5f12d-84d2-4a52-8802-d9e0381f4c90\") "
Oct 30 12:35:21.563833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.563776    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca" pod "7bb5f12d-84d2-4a52-8802-d9e0381f4c90" (UID: "7bb5f12d-84d2-4a52-8802-d9e0381f4c90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:21.564946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.564920    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq" pod "7bb5f12d-84d2-4a52-8802-d9e0381f4c90" (UID: "7bb5f12d-84d2-4a52-8802-d9e0381f4c90"). InnerVolumeSpecName "kube-api-access-76qpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:21.576639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.576605    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca" pod "7bb5f12d-84d2-4a52-8802-d9e0381f4c90" (UID: "7bb5f12d-84d2-4a52-8802-d9e0381f4c90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:21.645070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.645029    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"]
Oct 30 12:35:21.645785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.645761    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7bb5f12d-84d2-4a52-8802-d9e0381f4c90" containerName="revoke-keys"
Oct 30 12:35:21.645785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.645780    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb5f12d-84d2-4a52-8802-d9e0381f4c90" containerName="revoke-keys"
Oct 30 12:35:21.646016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.645981    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb5f12d-84d2-4a52-8802-d9e0381f4c90" containerName="revoke-keys"
Oct 30 12:35:21.649748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.649726    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.665271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.664978    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:21.665271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.665009    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:21.665271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.665024    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qpq\" (UniqueName: \"kubernetes.io/projected/7bb5f12d-84d2-4a52-8802-d9e0381f4c90-kube-api-access-76qpq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:21.670412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.670381    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"]
Oct 30 12:35:21.766033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.765941    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.766033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.765988    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.766250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.766191    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvxz\" (UniqueName: \"kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.867842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.867803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.868061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.867853    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.868061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.868015    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvxz\" (UniqueName: \"kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.868657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.868619    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.870560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.870522    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.891956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.891930    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvxz\" (UniqueName: \"kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz\") pod \"maas-api-revoke-sub-3d99c5b4-wrjxg\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:21.974698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:21.974664    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:22.074210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:22.074139    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:22.285538 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:35:22.285502    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eccb7d8_55a3_4405_b5be_2684d495c893.slice/crio-1aabf2889b43b43b25d13a57554c9a2682a006af4d87112818a45ce705126485 WatchSource:0}: Error finding container 1aabf2889b43b43b25d13a57554c9a2682a006af4d87112818a45ce705126485: Status 404 returned error can't find the container with id 1aabf2889b43b43b25d13a57554c9a2682a006af4d87112818a45ce705126485
Oct 30 12:35:22.287980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:22.287953    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"]
Oct 30 12:35:22.339962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:22.339928    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"
Oct 30 12:35:22.340155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:22.339964    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99" event={"ID":"7bb5f12d-84d2-4a52-8802-d9e0381f4c90","Type":"ContainerDied","Data":"12d221e811a7d257a094d3ec7823ce5cf5145a036d91082289dfdaa05405b22e"}
Oct 30 12:35:22.340155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:22.340011    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d221e811a7d257a094d3ec7823ce5cf5145a036d91082289dfdaa05405b22e"
Oct 30 12:35:22.341672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:22.341643    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg" event={"ID":"5eccb7d8-55a3-4405-b5be-2684d495c893","Type":"ContainerStarted","Data":"1aabf2889b43b43b25d13a57554c9a2682a006af4d87112818a45ce705126485"}
Oct 30 12:35:23.298003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.297800    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"]
Oct 30 12:35:23.309941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.309876    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.315630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.315594    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"]
Oct 30 12:35:23.354715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.354666    5060 generic.go:356] "Generic (PLEG): container finished" podID="5eccb7d8-55a3-4405-b5be-2684d495c893" containerID="31542232f5fed8a42d876fd250feef789eafedc04f482f19840ce55d89a21907" exitCode=0
Oct 30 12:35:23.355017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.354993    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg" event={"ID":"5eccb7d8-55a3-4405-b5be-2684d495c893","Type":"ContainerDied","Data":"31542232f5fed8a42d876fd250feef789eafedc04f482f19840ce55d89a21907"}
Oct 30 12:35:23.394666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.394628    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcrv\" (UniqueName: \"kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.394666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.394671    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.394917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.394777    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.495926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.495889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.496145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.495999    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.496292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.496277    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcrv\" (UniqueName: \"kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.496842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.496759    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.498833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.498808    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.520903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.520868    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcrv\" (UniqueName: \"kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv\") pod \"maas-api-revoke-sub-fb7cde04-7xw2t\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.672736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.672644    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:23.848530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.848492    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:23.983736 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:35:23.983699    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda176d82f_2ad6_4bbe_9937_71214a1d9521.slice/crio-05acacb2603bc41207abcd834c0a651d53be63e312eb9554c701758468d3721b WatchSource:0}: Error finding container 05acacb2603bc41207abcd834c0a651d53be63e312eb9554c701758468d3721b: Status 404 returned error can't find the container with id 05acacb2603bc41207abcd834c0a651d53be63e312eb9554c701758468d3721b
Oct 30 12:35:23.986727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:23.986686    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"]
Oct 30 12:35:24.365545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.365442    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t" event={"ID":"a176d82f-2ad6-4bbe-9937-71214a1d9521","Type":"ContainerStarted","Data":"05acacb2603bc41207abcd834c0a651d53be63e312eb9554c701758468d3721b"}
Oct 30 12:35:24.543664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.543633    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:24.713098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.713062    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca\") pod \"5eccb7d8-55a3-4405-b5be-2684d495c893\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") "
Oct 30 12:35:24.713098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.713104    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz\" (UniqueName: \"kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz\") pod \"5eccb7d8-55a3-4405-b5be-2684d495c893\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") "
Oct 30 12:35:24.713369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.713242    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca\") pod \"5eccb7d8-55a3-4405-b5be-2684d495c893\" (UID: \"5eccb7d8-55a3-4405-b5be-2684d495c893\") "
Oct 30 12:35:24.713435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.713411    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca" pod "5eccb7d8-55a3-4405-b5be-2684d495c893" (UID: "5eccb7d8-55a3-4405-b5be-2684d495c893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:24.713935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.713916    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.715529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.715503    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca" pod "5eccb7d8-55a3-4405-b5be-2684d495c893" (UID: "5eccb7d8-55a3-4405-b5be-2684d495c893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:24.715663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.715573    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz" pod "5eccb7d8-55a3-4405-b5be-2684d495c893" (UID: "5eccb7d8-55a3-4405-b5be-2684d495c893"). InnerVolumeSpecName "kube-api-access-psvxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:24.815228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.815193    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvxz\" (UniqueName: \"kubernetes.io/projected/5eccb7d8-55a3-4405-b5be-2684d495c893-kube-api-access-psvxz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.815228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:24.815225    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccb7d8-55a3-4405-b5be-2684d495c893-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:25.376184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:25.376091    5060 generic.go:356] "Generic (PLEG): container finished" podID="a176d82f-2ad6-4bbe-9937-71214a1d9521" containerID="0ddc0ad463ed30e86aead1487d19ced67cff18d25c747f9210ef2f184499ee02" exitCode=0
Oct 30 12:35:25.376670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:25.376175    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t" event={"ID":"a176d82f-2ad6-4bbe-9937-71214a1d9521","Type":"ContainerDied","Data":"0ddc0ad463ed30e86aead1487d19ced67cff18d25c747f9210ef2f184499ee02"}
Oct 30 12:35:25.377783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:25.377758    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg" event={"ID":"5eccb7d8-55a3-4405-b5be-2684d495c893","Type":"ContainerDied","Data":"1aabf2889b43b43b25d13a57554c9a2682a006af4d87112818a45ce705126485"}
Oct 30 12:35:25.377783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:25.377774    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"
Oct 30 12:35:25.377957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:25.377797    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aabf2889b43b43b25d13a57554c9a2682a006af4d87112818a45ce705126485"
Oct 30 12:35:25.659367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:25.659264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:26.575844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.575816    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:26.742484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.742432    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca\") pod \"a176d82f-2ad6-4bbe-9937-71214a1d9521\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") "
Oct 30 12:35:26.742700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.742594    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca\") pod \"a176d82f-2ad6-4bbe-9937-71214a1d9521\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") "
Oct 30 12:35:26.742700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.742662    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv\" (UniqueName: \"kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv\") pod \"a176d82f-2ad6-4bbe-9937-71214a1d9521\" (UID: \"a176d82f-2ad6-4bbe-9937-71214a1d9521\") "
Oct 30 12:35:26.742917 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.742893    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca" pod "a176d82f-2ad6-4bbe-9937-71214a1d9521" (UID: "a176d82f-2ad6-4bbe-9937-71214a1d9521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:26.743277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.743258    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:26.745001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.744975    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca" pod "a176d82f-2ad6-4bbe-9937-71214a1d9521" (UID: "a176d82f-2ad6-4bbe-9937-71214a1d9521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:26.745001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.744992    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv" pod "a176d82f-2ad6-4bbe-9937-71214a1d9521" (UID: "a176d82f-2ad6-4bbe-9937-71214a1d9521"). InnerVolumeSpecName "kube-api-access-4tcrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:26.845148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.845105    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a176d82f-2ad6-4bbe-9937-71214a1d9521-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:26.845148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:26.845137    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcrv\" (UniqueName: \"kubernetes.io/projected/a176d82f-2ad6-4bbe-9937-71214a1d9521-kube-api-access-4tcrv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:27.376495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:27.368008    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:27.398542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:27.398492    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t" event={"ID":"a176d82f-2ad6-4bbe-9937-71214a1d9521","Type":"ContainerDied","Data":"05acacb2603bc41207abcd834c0a651d53be63e312eb9554c701758468d3721b"}
Oct 30 12:35:27.398542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:27.398520    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"
Oct 30 12:35:27.398829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:27.398556    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05acacb2603bc41207abcd834c0a651d53be63e312eb9554c701758468d3721b"
Oct 30 12:35:28.580391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:28.580355    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:28.719626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:28.719584    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:29.074802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:29.074765    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:30.760483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:30.760415    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:32.456445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:32.456411    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:32.650125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:32.650089    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j" podUID="c0d00dad-63da-4d02-9fd1-a80b159a44cb" containerName="main" containerID="cri-o://11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512" gracePeriod=45
Oct 30 12:35:32.960931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:32.960904    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:35:33.042809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.042720    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs\") pod \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") "
Oct 30 12:35:33.042965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.042820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache\") pod \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") "
Oct 30 12:35:33.042965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.042861    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp\" (UniqueName: \"kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp\") pod \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") "
Oct 30 12:35:33.043059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043019    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home\") pod \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") "
Oct 30 12:35:33.043171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir\") pod \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") "
Oct 30 12:35:33.043243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043201    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm\") pod \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\" (UID: \"c0d00dad-63da-4d02-9fd1-a80b159a44cb\") "
Oct 30 12:35:33.043243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043201    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache" pod "c0d00dad-63da-4d02-9fd1-a80b159a44cb" (UID: "c0d00dad-63da-4d02-9fd1-a80b159a44cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:33.043349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043248    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home" pod "c0d00dad-63da-4d02-9fd1-a80b159a44cb" (UID: "c0d00dad-63da-4d02-9fd1-a80b159a44cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:33.043431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043407    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir" pod "c0d00dad-63da-4d02-9fd1-a80b159a44cb" (UID: "c0d00dad-63da-4d02-9fd1-a80b159a44cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:33.043856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043830    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.043995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043859    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.043995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.043875    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.046134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.046115    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs" pod "c0d00dad-63da-4d02-9fd1-a80b159a44cb" (UID: "c0d00dad-63da-4d02-9fd1-a80b159a44cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:33.046267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.046246    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp" pod "c0d00dad-63da-4d02-9fd1-a80b159a44cb" (UID: "c0d00dad-63da-4d02-9fd1-a80b159a44cb"). InnerVolumeSpecName "kube-api-access-lxnsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:33.050787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.050752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm" pod "c0d00dad-63da-4d02-9fd1-a80b159a44cb" (UID: "c0d00dad-63da-4d02-9fd1-a80b159a44cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:33.146022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.145986    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnsp\" (UniqueName: \"kubernetes.io/projected/c0d00dad-63da-4d02-9fd1-a80b159a44cb-kube-api-access-lxnsp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.146022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.146019    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d00dad-63da-4d02-9fd1-a80b159a44cb-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.146251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.146041    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d00dad-63da-4d02-9fd1-a80b159a44cb-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.463147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.463115    5060 generic.go:356] "Generic (PLEG): container finished" podID="c0d00dad-63da-4d02-9fd1-a80b159a44cb" containerID="11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512" exitCode=0
Oct 30 12:35:33.463692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.463169    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"
Oct 30 12:35:33.463692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.463206    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j" event={"ID":"c0d00dad-63da-4d02-9fd1-a80b159a44cb","Type":"ContainerDied","Data":"11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512"}
Oct 30 12:35:33.463692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.463251    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j" event={"ID":"c0d00dad-63da-4d02-9fd1-a80b159a44cb","Type":"ContainerDied","Data":"873d0454856db02a15fc74872b616d4e5a57ac9a4506a138ed6f4f1bdb9fa0ea"}
Oct 30 12:35:33.463692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.463272    5060 scope.go:122] "RemoveContainer" containerID="11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512"
Oct 30 12:35:33.475628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.475609    5060 scope.go:122] "RemoveContainer" containerID="11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512"
Oct 30 12:35:33.475895 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:35:33.475876    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512\": container with ID starting with 11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512 not found: ID does not exist" containerID="11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512"
Oct 30 12:35:33.475942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.475907    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512"} err="failed to get container status \"11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512\": rpc error: code = NotFound desc = could not find container \"11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512\": container with ID starting with 11303c2dd09cdf43d5c6ce252417e9df0f6a35c36485342a124563fe98a05512 not found: ID does not exist"
Oct 30 12:35:33.521915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.521879    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"]
Oct 30 12:35:33.527248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.527220    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dd0ff7/test-model-dd0ff7-kserve-6f5d564d69-qdh2j"]
Oct 30 12:35:33.683168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:33.683134    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:34.220072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:34.220035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:34.336986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:34.336948    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d00dad-63da-4d02-9fd1-a80b159a44cb" path="/var/lib/kubelet/pods/c0d00dad-63da-4d02-9fd1-a80b159a44cb/volumes"
Oct 30 12:35:35.922147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:35.922108    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:37.657838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:37.657790    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:39.077620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.077586    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"]
Oct 30 12:35:39.134722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.134686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:39.493563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.491281    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:39.693786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.693749    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"]
Oct 30 12:35:39.694717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694684    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0d00dad-63da-4d02-9fd1-a80b159a44cb" containerName="main"
Oct 30 12:35:39.694717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694713    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d00dad-63da-4d02-9fd1-a80b159a44cb" containerName="main"
Oct 30 12:35:39.694902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694754    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a176d82f-2ad6-4bbe-9937-71214a1d9521" containerName="revoke-keys"
Oct 30 12:35:39.694902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694762    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a176d82f-2ad6-4bbe-9937-71214a1d9521" containerName="revoke-keys"
Oct 30 12:35:39.694902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694811    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5eccb7d8-55a3-4405-b5be-2684d495c893" containerName="revoke-keys"
Oct 30 12:35:39.694902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694818    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eccb7d8-55a3-4405-b5be-2684d495c893" containerName="revoke-keys"
Oct 30 12:35:39.695041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.694988    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a176d82f-2ad6-4bbe-9937-71214a1d9521" containerName="revoke-keys"
Oct 30 12:35:39.695041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.695007    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eccb7d8-55a3-4405-b5be-2684d495c893" containerName="revoke-keys"
Oct 30 12:35:39.695041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.695024    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d00dad-63da-4d02-9fd1-a80b159a44cb" containerName="main"
Oct 30 12:35:39.700915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.700889    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.724416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.724367    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"]
Oct 30 12:35:39.725135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.725107    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:35:39.725285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.725168    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:35:39.753723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.753637    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdn54\" (UniqueName: \"kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.753960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.753939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.754021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.753975    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.786237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.786213    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:35:39.786421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.786401    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:35:39.855180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.855142    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdn54\" (UniqueName: \"kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.855382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.855361    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.855435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.855396    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.856012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.855988    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.858017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.857994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:39.878826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:39.878806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdn54\" (UniqueName: \"kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54\") pod \"maas-api-revoke-sub-7318ab2b-ktvzs\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:40.019511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:40.019368    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:40.335500 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:35:40.335398    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b291a3_5569_4ce5_bc03_85d0e15e9893.slice/crio-4bb4cfa432a25262d5854e83f18ba8caef77f9f87de1fc16eaa8fc233ce6fd71 WatchSource:0}: Error finding container 4bb4cfa432a25262d5854e83f18ba8caef77f9f87de1fc16eaa8fc233ce6fd71: Status 404 returned error can't find the container with id 4bb4cfa432a25262d5854e83f18ba8caef77f9f87de1fc16eaa8fc233ce6fd71
Oct 30 12:35:40.340162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:40.340137    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"]
Oct 30 12:35:40.601546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:40.600695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs" event={"ID":"31b291a3-5569-4ce5-bc03-85d0e15e9893","Type":"ContainerStarted","Data":"4bb4cfa432a25262d5854e83f18ba8caef77f9f87de1fc16eaa8fc233ce6fd71"}
Oct 30 12:35:41.065588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:41.064363    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:41.366078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:41.366040    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:41.616619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:41.616511    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs" event={"ID":"31b291a3-5569-4ce5-bc03-85d0e15e9893","Type":"ContainerStarted","Data":"2f135f0dbecaccabf4fc449dcba4f211531355bbbe320d998fac147011c652b2"}
Oct 30 12:35:41.666500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:41.666427    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs" podStartSLOduration=2.666413355 podStartE2EDuration="2.666413355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:41.663121485 +0000 UTC m=+3945.458006006" watchObservedRunningTime="2026-10-30 12:35:41.666413355 +0000 UTC m=+3945.461297848"
Oct 30 12:35:42.670513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:42.670152    5060 generic.go:356] "Generic (PLEG): container finished" podID="31b291a3-5569-4ce5-bc03-85d0e15e9893" containerID="2f135f0dbecaccabf4fc449dcba4f211531355bbbe320d998fac147011c652b2" exitCode=0
Oct 30 12:35:42.670513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:42.670437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs" event={"ID":"31b291a3-5569-4ce5-bc03-85d0e15e9893","Type":"ContainerDied","Data":"2f135f0dbecaccabf4fc449dcba4f211531355bbbe320d998fac147011c652b2"}
Oct 30 12:35:43.062756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.062657    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"]
Oct 30 12:35:43.076170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.076122    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca070d4d-2rs87"]
Oct 30 12:35:43.723001 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.722957    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:43.896632 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.896600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:43.933304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.933209    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca\") pod \"31b291a3-5569-4ce5-bc03-85d0e15e9893\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") "
Oct 30 12:35:43.933587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.933314    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca\") pod \"31b291a3-5569-4ce5-bc03-85d0e15e9893\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") "
Oct 30 12:35:43.933587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.933465    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54\" (UniqueName: \"kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54\") pod \"31b291a3-5569-4ce5-bc03-85d0e15e9893\" (UID: \"31b291a3-5569-4ce5-bc03-85d0e15e9893\") "
Oct 30 12:35:43.933731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.933690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca" pod "31b291a3-5569-4ce5-bc03-85d0e15e9893" (UID: "31b291a3-5569-4ce5-bc03-85d0e15e9893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:43.934128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.934099    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:43.936275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.936229    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca" pod "31b291a3-5569-4ce5-bc03-85d0e15e9893" (UID: "31b291a3-5569-4ce5-bc03-85d0e15e9893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:43.940013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:43.939969    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54" pod "31b291a3-5569-4ce5-bc03-85d0e15e9893" (UID: "31b291a3-5569-4ce5-bc03-85d0e15e9893"). InnerVolumeSpecName "kube-api-access-fdn54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:44.035599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:44.035555    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b291a3-5569-4ce5-bc03-85d0e15e9893-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:44.035599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:44.035599    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdn54\" (UniqueName: \"kubernetes.io/projected/31b291a3-5569-4ce5-bc03-85d0e15e9893-kube-api-access-fdn54\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:44.336624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:44.336587    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b748546-4ce6-461f-ae9a-5788fd038e6e" path="/var/lib/kubelet/pods/6b748546-4ce6-461f-ae9a-5788fd038e6e/volumes"
Oct 30 12:35:44.695032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:44.694934    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs" event={"ID":"31b291a3-5569-4ce5-bc03-85d0e15e9893","Type":"ContainerDied","Data":"4bb4cfa432a25262d5854e83f18ba8caef77f9f87de1fc16eaa8fc233ce6fd71"}
Oct 30 12:35:44.695032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:44.694975    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"
Oct 30 12:35:44.695032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:44.694988    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bb4cfa432a25262d5854e83f18ba8caef77f9f87de1fc16eaa8fc233ce6fd71"
Oct 30 12:35:45.385521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:45.378021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:47.029826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:47.029785    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:48.745940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:48.745904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:49.858152 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:35:49.858118    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:35:50.365109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:50.365072    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:51.422170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:51.422124    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.144908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.144879    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.550596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.550554    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"]
Oct 30 12:35:52.620483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.620417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.824127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.824025    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"]
Oct 30 12:35:52.825121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.825093    5060 cpu_manager.go:414] "containerMap: removing container" podUID="31b291a3-5569-4ce5-bc03-85d0e15e9893" containerName="revoke-keys"
Oct 30 12:35:52.825121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.825119    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b291a3-5569-4ce5-bc03-85d0e15e9893" containerName="revoke-keys"
Oct 30 12:35:52.825331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.825302    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b291a3-5569-4ce5-bc03-85d0e15e9893" containerName="revoke-keys"
Oct 30 12:35:52.830996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.830973    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.845933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.845902    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:35:52.846114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.846080    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:35:52.846335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.846309    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"]
Oct 30 12:35:52.871147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.871114    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:35:52.871147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.871145    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:35:52.882708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.882593    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.883303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.883253    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x469j\" (UniqueName: \"kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.883657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.883604    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.985171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.985119    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.985381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.985228    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x469j\" (UniqueName: \"kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.985381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.985298    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.986174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.986147    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:52.988307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:52.988278    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:53.011133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:53.011101    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x469j\" (UniqueName: \"kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j\") pod \"maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:53.148090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:53.148004    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:53.459029 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:35:53.458995    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e3549a_acf7_42a1_961f_27b00ab0a0c2.slice/crio-6e9b1cb86d0d5e9d1eb546c375d6326afd2babd576095a9de7da974dcb8261ed WatchSource:0}: Error finding container 6e9b1cb86d0d5e9d1eb546c375d6326afd2babd576095a9de7da974dcb8261ed: Status 404 returned error can't find the container with id 6e9b1cb86d0d5e9d1eb546c375d6326afd2babd576095a9de7da974dcb8261ed
Oct 30 12:35:53.461557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:53.461535    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"]
Oct 30 12:35:53.797322 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:53.797233    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst" event={"ID":"03e3549a-acf7-42a1-961f-27b00ab0a0c2","Type":"ContainerStarted","Data":"6e9b1cb86d0d5e9d1eb546c375d6326afd2babd576095a9de7da974dcb8261ed"}
Oct 30 12:35:54.079098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.079004    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:54.116477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.115949    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" podUID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" containerName="main" containerID="cri-o://02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97" gracePeriod=45
Oct 30 12:35:54.200022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.199977    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" podUID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" containerName="main" probeResult="failure" output="Get \"https://10.132.1.133:8000/ready\": dial tcp 10.132.1.133:8000: connect: connection refused"
Oct 30 12:35:54.398443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.398416    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:35:54.513782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.513746    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx\" (UniqueName: \"kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx\") pod \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") "
Oct 30 12:35:54.514139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514116    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir\") pod \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") "
Oct 30 12:35:54.514245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514191    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache\") pod \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") "
Oct 30 12:35:54.514315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514287    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm\") pod \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") "
Oct 30 12:35:54.514507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514346    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home\") pod \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") "
Oct 30 12:35:54.514507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514421    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir" pod "e3485463-ad00-456d-8b5f-8f6d09ef5c35" (UID: "e3485463-ad00-456d-8b5f-8f6d09ef5c35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:54.514507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514474    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs\") pod \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\" (UID: \"e3485463-ad00-456d-8b5f-8f6d09ef5c35\") "
Oct 30 12:35:54.514694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514635    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache" pod "e3485463-ad00-456d-8b5f-8f6d09ef5c35" (UID: "e3485463-ad00-456d-8b5f-8f6d09ef5c35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:54.514844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.514822    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home" pod "e3485463-ad00-456d-8b5f-8f6d09ef5c35" (UID: "e3485463-ad00-456d-8b5f-8f6d09ef5c35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:54.515162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.515140    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.515411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.515167    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.515411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.515182    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.516351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.516327    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx" pod "e3485463-ad00-456d-8b5f-8f6d09ef5c35" (UID: "e3485463-ad00-456d-8b5f-8f6d09ef5c35"). InnerVolumeSpecName "kube-api-access-thxwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:54.516663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.516641    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm" pod "e3485463-ad00-456d-8b5f-8f6d09ef5c35" (UID: "e3485463-ad00-456d-8b5f-8f6d09ef5c35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:54.516936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.516910    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs" pod "e3485463-ad00-456d-8b5f-8f6d09ef5c35" (UID: "e3485463-ad00-456d-8b5f-8f6d09ef5c35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:54.616391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.616319    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3485463-ad00-456d-8b5f-8f6d09ef5c35-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.616391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.616347    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3485463-ad00-456d-8b5f-8f6d09ef5c35-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.616391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.616357    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thxwx\" (UniqueName: \"kubernetes.io/projected/e3485463-ad00-456d-8b5f-8f6d09ef5c35-kube-api-access-thxwx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.809158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.809127    5060 generic.go:356] "Generic (PLEG): container finished" podID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" containerID="02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97" exitCode=0
Oct 30 12:35:54.809662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.809232    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" event={"ID":"e3485463-ad00-456d-8b5f-8f6d09ef5c35","Type":"ContainerDied","Data":"02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97"}
Oct 30 12:35:54.809662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.809267    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"
Oct 30 12:35:54.809662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.809286    5060 scope.go:122] "RemoveContainer" containerID="02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97"
Oct 30 12:35:54.809662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.809272    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b" event={"ID":"e3485463-ad00-456d-8b5f-8f6d09ef5c35","Type":"ContainerDied","Data":"d4d08f3d2850751b7d10a8eea7dc8b243535abf0dc3ab4ef4f82b4fffdae7f3b"}
Oct 30 12:35:54.811584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.811553    5060 generic.go:356] "Generic (PLEG): container finished" podID="03e3549a-acf7-42a1-961f-27b00ab0a0c2" containerID="1542ebcd4d41851bdf107f941e88b7b2346ef7f6128714f090e10175bc7015ea" exitCode=0
Oct 30 12:35:54.811756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.811707    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst" event={"ID":"03e3549a-acf7-42a1-961f-27b00ab0a0c2","Type":"ContainerDied","Data":"1542ebcd4d41851bdf107f941e88b7b2346ef7f6128714f090e10175bc7015ea"}
Oct 30 12:35:54.822555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.822540    5060 scope.go:122] "RemoveContainer" containerID="02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97"
Oct 30 12:35:54.822855 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:35:54.822835    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97\": container with ID starting with 02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97 not found: ID does not exist" containerID="02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97"
Oct 30 12:35:54.822931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.822862    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97"} err="failed to get container status \"02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97\": rpc error: code = NotFound desc = could not find container \"02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97\": container with ID starting with 02b596c8baf94a2232aaa82a4d8569536f999608db75904afbd84d6d91543f97 not found: ID does not exist"
Oct 30 12:35:54.863833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.863804    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"]
Oct 30 12:35:54.868607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.868542    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-31a6e3/test-model-31a6e3-kserve-68b6645b77-9v84b"]
Oct 30 12:35:54.988331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:54.988296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:55.519487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:55.515977    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:55.708477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:55.707893    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:55.991800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:55.991735    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:56.132173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.132100    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:56.137481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.137436    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:56.246124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.246084    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca\") pod \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") "
Oct 30 12:35:56.246310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.246205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j\" (UniqueName: \"kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j\") pod \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") "
Oct 30 12:35:56.246357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.246324    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca\") pod \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\" (UID: \"03e3549a-acf7-42a1-961f-27b00ab0a0c2\") "
Oct 30 12:35:56.246708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.246682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca" pod "03e3549a-acf7-42a1-961f-27b00ab0a0c2" (UID: "03e3549a-acf7-42a1-961f-27b00ab0a0c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:56.247056 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.247038    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:56.248528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.248501    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca" pod "03e3549a-acf7-42a1-961f-27b00ab0a0c2" (UID: "03e3549a-acf7-42a1-961f-27b00ab0a0c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:35:56.248528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.248512    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j" pod "03e3549a-acf7-42a1-961f-27b00ab0a0c2" (UID: "03e3549a-acf7-42a1-961f-27b00ab0a0c2"). InnerVolumeSpecName "kube-api-access-x469j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:56.276315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.276282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:56.340391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.340357    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" path="/var/lib/kubelet/pods/e3485463-ad00-456d-8b5f-8f6d09ef5c35/volumes"
Oct 30 12:35:56.347880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.347853    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e3549a-acf7-42a1-961f-27b00ab0a0c2-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:56.347880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.347878    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x469j\" (UniqueName: \"kubernetes.io/projected/03e3549a-acf7-42a1-961f-27b00ab0a0c2-kube-api-access-x469j\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:35:56.872055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.872022    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"
Oct 30 12:35:56.872238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.872023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst" event={"ID":"03e3549a-acf7-42a1-961f-27b00ab0a0c2","Type":"ContainerDied","Data":"6e9b1cb86d0d5e9d1eb546c375d6326afd2babd576095a9de7da974dcb8261ed"}
Oct 30 12:35:56.872238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:56.872165    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9b1cb86d0d5e9d1eb546c375d6326afd2babd576095a9de7da974dcb8261ed"
Oct 30 12:35:57.677775 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:57.677727    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:59.453983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:59.453945    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:59.712106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:35:59.711996    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:01.163670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:01.163637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:01.323015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:01.322970    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:01.804655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:01.804615    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:03.070265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:03.070230    5060 scope.go:122] "RemoveContainer" containerID="91aecbc407b6f8a2951f385f08083e3e6a4bc6e6eb2a8c8abfb8b1ba68b8c275"
Oct 30 12:36:03.085496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:03.085306    5060 scope.go:122] "RemoveContainer" containerID="7bd8aaa51c2d42961273244d1f9212e78a5014916405ad2a50c5c99abc719df3"
Oct 30 12:36:03.100178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:03.100152    5060 scope.go:122] "RemoveContainer" containerID="bef0371641582458e891b8ee2af2fe86d676666e79deec385d03e72ba2363086"
Oct 30 12:36:03.516110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:03.516072    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:03.654916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:03.654880    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:05.286085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:05.286041    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:07.095006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.094967    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:07.576269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.576230    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2" podUID="3f1c88ae-0a86-4804-8393-9942a3ee2916" containerName="main" containerID="cri-o://842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5" gracePeriod=45
Oct 30 12:36:07.877505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.877433    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:36:07.965438 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965402    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs\") pod \"3f1c88ae-0a86-4804-8393-9942a3ee2916\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") "
Oct 30 12:36:07.965664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965560    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home\") pod \"3f1c88ae-0a86-4804-8393-9942a3ee2916\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") "
Oct 30 12:36:07.965664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965585    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm\") pod \"3f1c88ae-0a86-4804-8393-9942a3ee2916\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") "
Oct 30 12:36:07.965664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965626    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45\" (UniqueName: \"kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45\") pod \"3f1c88ae-0a86-4804-8393-9942a3ee2916\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") "
Oct 30 12:36:07.965869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965667    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir\") pod \"3f1c88ae-0a86-4804-8393-9942a3ee2916\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") "
Oct 30 12:36:07.965869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965736    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache\") pod \"3f1c88ae-0a86-4804-8393-9942a3ee2916\" (UID: \"3f1c88ae-0a86-4804-8393-9942a3ee2916\") "
Oct 30 12:36:07.965869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.965830    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home" pod "3f1c88ae-0a86-4804-8393-9942a3ee2916" (UID: "3f1c88ae-0a86-4804-8393-9942a3ee2916"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:07.966227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.966197    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache" pod "3f1c88ae-0a86-4804-8393-9942a3ee2916" (UID: "3f1c88ae-0a86-4804-8393-9942a3ee2916"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:07.966366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.966221    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir" pod "3f1c88ae-0a86-4804-8393-9942a3ee2916" (UID: "3f1c88ae-0a86-4804-8393-9942a3ee2916"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:07.966434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.966385    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:07.966434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.966408    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:07.966434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.966422    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:07.968250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.968223    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45" pod "3f1c88ae-0a86-4804-8393-9942a3ee2916" (UID: "3f1c88ae-0a86-4804-8393-9942a3ee2916"). InnerVolumeSpecName "kube-api-access-srh45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:07.968600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.968568    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm" pod "3f1c88ae-0a86-4804-8393-9942a3ee2916" (UID: "3f1c88ae-0a86-4804-8393-9942a3ee2916"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:07.968715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.968611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs" pod "3f1c88ae-0a86-4804-8393-9942a3ee2916" (UID: "3f1c88ae-0a86-4804-8393-9942a3ee2916"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:36:07.982494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.979428    5060 generic.go:356] "Generic (PLEG): container finished" podID="3f1c88ae-0a86-4804-8393-9942a3ee2916" containerID="842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5" exitCode=0
Oct 30 12:36:07.982494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.979682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2" event={"ID":"3f1c88ae-0a86-4804-8393-9942a3ee2916","Type":"ContainerDied","Data":"842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5"}
Oct 30 12:36:07.982494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.979727    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2" event={"ID":"3f1c88ae-0a86-4804-8393-9942a3ee2916","Type":"ContainerDied","Data":"bfdfa090a9b8eaf3c513b0aa69f3fcd94b1b80f1aa5585b3027ba4a178aa1d93"}
Oct 30 12:36:07.982494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.979750    5060 scope.go:122] "RemoveContainer" containerID="842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5"
Oct 30 12:36:07.982494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:07.979904    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"
Oct 30 12:36:08.007643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.007616    5060 scope.go:122] "RemoveContainer" containerID="842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5"
Oct 30 12:36:08.008021 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:36:08.007993    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5\": container with ID starting with 842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5 not found: ID does not exist" containerID="842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5"
Oct 30 12:36:08.008111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.008034    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5"} err="failed to get container status \"842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5\": rpc error: code = NotFound desc = could not find container \"842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5\": container with ID starting with 842b5454925b643f529dd6e870172a9cae9c972b0dad00a106a2c4523bfe8db5 not found: ID does not exist"
Oct 30 12:36:08.041641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.041600    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"]
Oct 30 12:36:08.047040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.047006    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ffea51/test-model-ffea51-kserve-7774879f67-g49r2"]
Oct 30 12:36:08.067922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.067890    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1c88ae-0a86-4804-8393-9942a3ee2916-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:08.067922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.067922    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1c88ae-0a86-4804-8393-9942a3ee2916-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:08.068103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.067935    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srh45\" (UniqueName: \"kubernetes.io/projected/3f1c88ae-0a86-4804-8393-9942a3ee2916-kube-api-access-srh45\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:08.335856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.335822    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1c88ae-0a86-4804-8393-9942a3ee2916" path="/var/lib/kubelet/pods/3f1c88ae-0a86-4804-8393-9942a3ee2916/volumes"
Oct 30 12:36:08.847134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:08.847094    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:10.521861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:10.521822    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:12.081356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:12.081321    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:12.694784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:12.694746    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:12.832088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:12.832049    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:12.846507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:12.846480    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"]
Oct 30 12:36:12.871787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:12.871752    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0190d8-nrc99"]
Oct 30 12:36:12.987909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:12.987872    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:13.371257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:13.371136    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:14.340548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:14.340477    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb5f12d-84d2-4a52-8802-d9e0381f4c90" path="/var/lib/kubelet/pods/7bb5f12d-84d2-4a52-8802-d9e0381f4c90/volumes"
Oct 30 12:36:14.970916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:14.970878    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:17.193241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:17.193195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:18.958913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:18.958874    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:19.684656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:19.684616    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:19.832270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:19.832222    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:19.863149 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:36:19.863111    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:36:20.007584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.007548    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"]
Oct 30 12:36:20.008527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.008490    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy" containerID="cri-o://aa6272ac63accd8b06c57b6c2b3a77663a9b10b66e70563ddb614950893b6eec" gracePeriod=30
Oct 30 12:36:20.038120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.038084    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:20.656224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656184    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"]
Oct 30 12:36:20.656908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656892    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3f1c88ae-0a86-4804-8393-9942a3ee2916" containerName="main"
Oct 30 12:36:20.656908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656909    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1c88ae-0a86-4804-8393-9942a3ee2916" containerName="main"
Oct 30 12:36:20.657032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656927    5060 cpu_manager.go:414] "containerMap: removing container" podUID="03e3549a-acf7-42a1-961f-27b00ab0a0c2" containerName="revoke-keys"
Oct 30 12:36:20.657032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656932    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e3549a-acf7-42a1-961f-27b00ab0a0c2" containerName="revoke-keys"
Oct 30 12:36:20.657032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656951    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" containerName="main"
Oct 30 12:36:20.657032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.656956    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" containerName="main"
Oct 30 12:36:20.657176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.657066    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1c88ae-0a86-4804-8393-9942a3ee2916" containerName="main"
Oct 30 12:36:20.657176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.657076    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e3549a-acf7-42a1-961f-27b00ab0a0c2" containerName="revoke-keys"
Oct 30 12:36:20.657176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.657084    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3485463-ad00-456d-8b5f-8f6d09ef5c35" containerName="main"
Oct 30 12:36:20.662267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.662248    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.678186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.678150    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"]
Oct 30 12:36:20.685520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.685480    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:36:20.685697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.685532    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:36:20.685697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.685604    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:36:20.689777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.689746    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:36:20.712822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.712781    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.713050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.712914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.713050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.712985    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbcd\" (UniqueName: \"kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.798399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.798358    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:20.834574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.826048    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbcd\" (UniqueName: \"kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.834574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.826326    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.834574 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.826443    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.847699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.847655    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.849151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.849121    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.855657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.855619    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbcd\" (UniqueName: \"kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd\") pod \"maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:20.980006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:20.979967    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:21.296828 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:36:21.296790    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8ecb68_0066_42cc_adbf_77598d0eacc3.slice/crio-5cce28d243f6d626af4fcb33872d42a6c9d48d197071916069a19c61a6a69956 WatchSource:0}: Error finding container 5cce28d243f6d626af4fcb33872d42a6c9d48d197071916069a19c61a6a69956: Status 404 returned error can't find the container with id 5cce28d243f6d626af4fcb33872d42a6c9d48d197071916069a19c61a6a69956
Oct 30 12:36:21.298769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:21.298737    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"]
Oct 30 12:36:22.245175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:22.245133    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q" event={"ID":"ae8ecb68-0066-42cc-adbf-77598d0eacc3","Type":"ContainerStarted","Data":"5249e036ec8c06c85764363055ff70ea77fec3db2a0d705b12105a24af3789d7"}
Oct 30 12:36:22.245397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:22.245193    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q" event={"ID":"ae8ecb68-0066-42cc-adbf-77598d0eacc3","Type":"ContainerStarted","Data":"5cce28d243f6d626af4fcb33872d42a6c9d48d197071916069a19c61a6a69956"}
Oct 30 12:36:22.294981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:22.294923    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q" podStartSLOduration=2.294901883 podStartE2EDuration="2.294901883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:22.291041319 +0000 UTC m=+3986.085925816" watchObservedRunningTime="2026-10-30 12:36:22.294901883 +0000 UTC m=+3986.089786378"
Oct 30 12:36:22.548425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:22.548327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.256640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:23.256605    5060 generic.go:356] "Generic (PLEG): container finished" podID="ae8ecb68-0066-42cc-adbf-77598d0eacc3" containerID="5249e036ec8c06c85764363055ff70ea77fec3db2a0d705b12105a24af3789d7" exitCode=0
Oct 30 12:36:23.256849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:23.256700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q" event={"ID":"ae8ecb68-0066-42cc-adbf-77598d0eacc3","Type":"ContainerDied","Data":"5249e036ec8c06c85764363055ff70ea77fec3db2a0d705b12105a24af3789d7"}
Oct 30 12:36:24.294284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.294243    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:24.479520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.479492    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:24.487709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.487669    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd\" (UniqueName: \"kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd\") pod \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") "
Oct 30 12:36:24.488052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.488027    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca\") pod \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") "
Oct 30 12:36:24.488204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.488076    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca\") pod \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\" (UID: \"ae8ecb68-0066-42cc-adbf-77598d0eacc3\") "
Oct 30 12:36:24.488494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.488406    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca" pod "ae8ecb68-0066-42cc-adbf-77598d0eacc3" (UID: "ae8ecb68-0066-42cc-adbf-77598d0eacc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:24.488866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.488841    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:24.490161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.490133    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd" pod "ae8ecb68-0066-42cc-adbf-77598d0eacc3" (UID: "ae8ecb68-0066-42cc-adbf-77598d0eacc3"). InnerVolumeSpecName "kube-api-access-zmbcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:24.490580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.490555    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca" pod "ae8ecb68-0066-42cc-adbf-77598d0eacc3" (UID: "ae8ecb68-0066-42cc-adbf-77598d0eacc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:36:24.590613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.590522    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8ecb68-0066-42cc-adbf-77598d0eacc3-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:24.590613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:24.590558    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmbcd\" (UniqueName: \"kubernetes.io/projected/ae8ecb68-0066-42cc-adbf-77598d0eacc3-kube-api-access-zmbcd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.274258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.274229    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"
Oct 30 12:36:25.274473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.274227    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q" event={"ID":"ae8ecb68-0066-42cc-adbf-77598d0eacc3","Type":"ContainerDied","Data":"5cce28d243f6d626af4fcb33872d42a6c9d48d197071916069a19c61a6a69956"}
Oct 30 12:36:25.274473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.274418    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cce28d243f6d626af4fcb33872d42a6c9d48d197071916069a19c61a6a69956"
Oct 30 12:36:25.276412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.276387    5060 generic.go:356] "Generic (PLEG): container finished" podID="c0720467-4a23-48da-b74f-10af54693218" containerID="aa6272ac63accd8b06c57b6c2b3a77663a9b10b66e70563ddb614950893b6eec" exitCode=0
Oct 30 12:36:25.276561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.276492    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" event={"ID":"c0720467-4a23-48da-b74f-10af54693218","Type":"ContainerDied","Data":"aa6272ac63accd8b06c57b6c2b3a77663a9b10b66e70563ddb614950893b6eec"}
Oct 30 12:36:25.276561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.276551    5060 scope.go:122] "RemoveContainer" containerID="e4b08865f0d0f8c69d1076eabc6f6a0ae1cb13a84a663efb2c6f883ffbfdc756"
Oct 30 12:36:25.735503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.735439    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807230    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807316    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807432    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807732    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.807999    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.808144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.808236    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket\") pod \"c0720467-4a23-48da-b74f-10af54693218\" (UID: \"c0720467-4a23-48da-b74f-10af54693218\") "
Oct 30 12:36:25.809560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.809131    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:25.810228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.810078    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:25.810612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.810572    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:25.810810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.810783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:25.811214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.811192    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:25.814168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.814127    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:25.814673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.814641    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:36:25.815057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.815021    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:25.815823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.815788    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6" pod "c0720467-4a23-48da-b74f-10af54693218" (UID: "c0720467-4a23-48da-b74f-10af54693218"). InnerVolumeSpecName "kube-api-access-5fhk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:25.910635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910594    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910627    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910640    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0720467-4a23-48da-b74f-10af54693218-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910651    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910666    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0720467-4a23-48da-b74f-10af54693218-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910677    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910688    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fhk6\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-kube-api-access-5fhk6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910699    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0720467-4a23-48da-b74f-10af54693218-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:25.910910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:25.910714    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0720467-4a23-48da-b74f-10af54693218-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:26.085929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.085826    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:26.244811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.244480    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:26.290208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.290175    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"
Oct 30 12:36:26.290403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.290171    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g" event={"ID":"c0720467-4a23-48da-b74f-10af54693218","Type":"ContainerDied","Data":"17382168403ef481dba657ee6882430a654608f2f510f20509168103ee1a38c9"}
Oct 30 12:36:26.290403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.290326    5060 scope.go:122] "RemoveContainer" containerID="aa6272ac63accd8b06c57b6c2b3a77663a9b10b66e70563ddb614950893b6eec"
Oct 30 12:36:26.348061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.347975    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"]
Oct 30 12:36:26.361004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.360973    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0190d8-openshift-default-7b7b476b99-9n45g"]
Oct 30 12:36:26.597871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:26.597828    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.035601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:28.035564    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.337431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:28.337338    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0720467-4a23-48da-b74f-10af54693218" path="/var/lib/kubelet/pods/c0720467-4a23-48da-b74f-10af54693218/volumes"
Oct 30 12:36:29.701677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:29.701637    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:31.147603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:31.147562    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"]
Oct 30 12:36:31.284367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:31.284322    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:31.429502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:31.429368    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:31.631597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:31.631559    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:32.835308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:32.835258    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:32.975526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:32.975479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:33.309351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:33.309308    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:35.016561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.016519    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:35.112739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.112694    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"]
Oct 30 12:36:35.113490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113468    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy"
Oct 30 12:36:35.113490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113487    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy"
Oct 30 12:36:35.113704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113525    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy"
Oct 30 12:36:35.113704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113531    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy"
Oct 30 12:36:35.113704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113556    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ae8ecb68-0066-42cc-adbf-77598d0eacc3" containerName="revoke-keys"
Oct 30 12:36:35.113704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113562    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8ecb68-0066-42cc-adbf-77598d0eacc3" containerName="revoke-keys"
Oct 30 12:36:35.113704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113689    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8ecb68-0066-42cc-adbf-77598d0eacc3" containerName="revoke-keys"
Oct 30 12:36:35.113704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113703    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy"
Oct 30 12:36:35.114080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.113712    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0720467-4a23-48da-b74f-10af54693218" containerName="istio-proxy"
Oct 30 12:36:35.117629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.117603    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.143992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.143961    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"openshift-service-ca.crt\""
Oct 30 12:36:35.144220 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.144009    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"kube-root-ca.crt\""
Oct 30 12:36:35.150799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.150770    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"]
Oct 30 12:36:35.154274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.154248    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"default-dockercfg-rxvrs\""
Oct 30 12:36:35.154431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.154389    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f1b14f\"/\"sub-test-model-f1b14f-kserve-self-signed-certs\""
Oct 30 12:36:35.208650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.208608    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:35.283334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.283247    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.283554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.283387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.283554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.283427    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.283554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.283537    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.283768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.283561    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pwt\" (UniqueName: \"kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.283768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.283661    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.388405    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.388501    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.388556    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.388592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pwt\" (UniqueName: \"kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.388628    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.388819    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.389735    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.389973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.394474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.390195    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.399910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.399688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.418503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.416789    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.423315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.423275    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pwt\" (UniqueName: \"kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt\") pod \"sub-test-model-f1b14f-kserve-6996df5589-s68hm\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") " pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.449403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.449359    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:35.849016 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:36:35.848988    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a490272_861e_43c2_bd2e_7d4c6b415c39.slice/crio-89014713eb3ef6de669c1acd62636d11fe00178af43d48d88629d87433fe35ab WatchSource:0}: Error finding container 89014713eb3ef6de669c1acd62636d11fe00178af43d48d88629d87433fe35ab: Status 404 returned error can't find the container with id 89014713eb3ef6de669c1acd62636d11fe00178af43d48d88629d87433fe35ab
Oct 30 12:36:35.850663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:35.850585    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"]
Oct 30 12:36:36.148181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:36.148093    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerName="payload-pre-processing" containerID="cri-o://c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed" gracePeriod=25
Oct 30 12:36:36.427494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:36.427334    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" event={"ID":"8a490272-861e-43c2-bd2e-7d4c6b415c39","Type":"ContainerStarted","Data":"2094b4465f160ebac725d2008dfec123b8a17d2f4100e8d59f3a0f76a5df33b8"}
Oct 30 12:36:36.427494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:36.427381    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" event={"ID":"8a490272-861e-43c2-bd2e-7d4c6b415c39","Type":"ContainerStarted","Data":"89014713eb3ef6de669c1acd62636d11fe00178af43d48d88629d87433fe35ab"}
Oct 30 12:36:36.450748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:36.450719    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:36.483091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:36.483020    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" podStartSLOduration=1.483000147 podStartE2EDuration="1.483000147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:36.475361259 +0000 UTC m=+4000.270245753" watchObservedRunningTime="2026-10-30 12:36:36.483000147 +0000 UTC m=+4000.277884644"
Oct 30 12:36:36.797607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:36.797549    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:38.580476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:38.573022    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:39.991032 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:39.990992    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:40.529289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:40.529252    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:40.665279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:40.665243    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"]
Oct 30 12:36:40.678437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:40.678390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:40.679421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:40.679392    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31a6e3-djwst"]
Oct 30 12:36:40.824840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:40.824750    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:41.185162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:41.185057    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:41.412175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:41.412139    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:36:42.336555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:42.336517    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e3549a-acf7-42a1-961f-27b00ab0a0c2" path="/var/lib/kubelet/pods/03e3549a-acf7-42a1-961f-27b00ab0a0c2/volumes"
Oct 30 12:36:42.561636 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:42.561597    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:44.432893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:44.432746    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:45.451432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:45.451397    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:45.464153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:45.464132    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:45.569569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:45.569541    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:36:46.093965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:46.093928    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:47.824913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:47.824789    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:47.975476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:47.974541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:48.148378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:48.148259    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:48.288841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:48.288802    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:48.524171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:48.524129    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:49.117368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.117273    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"]
Oct 30 12:36:49.124373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.124347    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.137751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.137721    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"]
Oct 30 12:36:49.139659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.139635    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:36:49.143569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.143548    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:36:49.224906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.224872    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:36:49.225116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.225004    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:36:49.258986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.258947    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.259228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.259206    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.259286 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.259268    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc47s\" (UniqueName: \"kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.360901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.360866    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.361086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.360928    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc47s\" (UniqueName: \"kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.361086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.361003    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.361768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.361745    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.363558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.363534    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.385189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.385119    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc47s\" (UniqueName: \"kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s\") pod \"maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.443150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.443121    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:49.758784 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:36:49.758739    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946659b4_7354_4e88_8e7b_13b8899bc0ed.slice/crio-0e6bb958bdc3b6a339cf28db5d938cf797f13c4df7ba6523db4efd19d6e834c3 WatchSource:0}: Error finding container 0e6bb958bdc3b6a339cf28db5d938cf797f13c4df7ba6523db4efd19d6e834c3: Status 404 returned error can't find the container with id 0e6bb958bdc3b6a339cf28db5d938cf797f13c4df7ba6523db4efd19d6e834c3
Oct 30 12:36:49.761665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.761631    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"]
Oct 30 12:36:49.856973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:49.856941    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:49.857211 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:36:49.857185    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:36:50.672753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:50.672715    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7" event={"ID":"946659b4-7354-4e88-8e7b-13b8899bc0ed","Type":"ContainerStarted","Data":"aeb3ef084162642a7d7885d58a05c169ca2bc4b1bcd84072f2a48cb47a645775"}
Oct 30 12:36:50.673249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:50.672775    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7" event={"ID":"946659b4-7354-4e88-8e7b-13b8899bc0ed","Type":"ContainerStarted","Data":"0e6bb958bdc3b6a339cf28db5d938cf797f13c4df7ba6523db4efd19d6e834c3"}
Oct 30 12:36:50.720591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:50.720518    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7" podStartSLOduration=1.7204985879999999 podStartE2EDuration="1.720498588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:50.713075523 +0000 UTC m=+4014.507960019" watchObservedRunningTime="2026-10-30 12:36:50.720498588 +0000 UTC m=+4014.515383083"
Oct 30 12:36:51.412332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:51.412298    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:36:51.684841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:51.684742    5060 generic.go:356] "Generic (PLEG): container finished" podID="946659b4-7354-4e88-8e7b-13b8899bc0ed" containerID="aeb3ef084162642a7d7885d58a05c169ca2bc4b1bcd84072f2a48cb47a645775" exitCode=0
Oct 30 12:36:51.685241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:51.684841    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7" event={"ID":"946659b4-7354-4e88-8e7b-13b8899bc0ed","Type":"ContainerDied","Data":"aeb3ef084162642a7d7885d58a05c169ca2bc4b1bcd84072f2a48cb47a645775"}
Oct 30 12:36:51.917983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:51.917946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:52.871190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:52.871160    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:53.020491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.020419    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca\") pod \"946659b4-7354-4e88-8e7b-13b8899bc0ed\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") "
Oct 30 12:36:53.020714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.020615    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca\") pod \"946659b4-7354-4e88-8e7b-13b8899bc0ed\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") "
Oct 30 12:36:53.020837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.020821    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s\" (UniqueName: \"kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s\") pod \"946659b4-7354-4e88-8e7b-13b8899bc0ed\" (UID: \"946659b4-7354-4e88-8e7b-13b8899bc0ed\") "
Oct 30 12:36:53.020916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.020850    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca" pod "946659b4-7354-4e88-8e7b-13b8899bc0ed" (UID: "946659b4-7354-4e88-8e7b-13b8899bc0ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:53.021607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.021579    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.023046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.023012    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca" pod "946659b4-7354-4e88-8e7b-13b8899bc0ed" (UID: "946659b4-7354-4e88-8e7b-13b8899bc0ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:36:53.023197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.023174    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s" pod "946659b4-7354-4e88-8e7b-13b8899bc0ed" (UID: "946659b4-7354-4e88-8e7b-13b8899bc0ed"). InnerVolumeSpecName "kube-api-access-rc47s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:53.123251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.123212    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946659b4-7354-4e88-8e7b-13b8899bc0ed-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.123251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.123250    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc47s\" (UniqueName: \"kubernetes.io/projected/946659b4-7354-4e88-8e7b-13b8899bc0ed-kube-api-access-rc47s\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.181093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.181023    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:53.334933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.334827    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:53.707272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.707235    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"
Oct 30 12:36:53.707272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.707256    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7" event={"ID":"946659b4-7354-4e88-8e7b-13b8899bc0ed","Type":"ContainerDied","Data":"0e6bb958bdc3b6a339cf28db5d938cf797f13c4df7ba6523db4efd19d6e834c3"}
Oct 30 12:36:53.707554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.707309    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6bb958bdc3b6a339cf28db5d938cf797f13c4df7ba6523db4efd19d6e834c3"
Oct 30 12:36:53.764603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:53.764554    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:55.455484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:55.455413    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:56.127936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.127902    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:56.365936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.365893    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:56.466006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.465979    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:36:56.588240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.588203    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj\" (UniqueName: \"kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj\") pod \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") "
Oct 30 12:36:56.588485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.588275    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume\") pod \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\" (UID: \"e8b2bceb-8b85-4d64-a4ee-03a590df1fed\") "
Oct 30 12:36:56.588822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.588793    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume" pod "e8b2bceb-8b85-4d64-a4ee-03a590df1fed" (UID: "e8b2bceb-8b85-4d64-a4ee-03a590df1fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:56.589128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.589101    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:56.590779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.590752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj" pod "e8b2bceb-8b85-4d64-a4ee-03a590df1fed" (UID: "e8b2bceb-8b85-4d64-a4ee-03a590df1fed"). InnerVolumeSpecName "kube-api-access-lt2tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:56.690889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.690811    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2tj\" (UniqueName: \"kubernetes.io/projected/e8b2bceb-8b85-4d64-a4ee-03a590df1fed-kube-api-access-lt2tj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:36:56.741383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.741351    5060 generic.go:356] "Generic (PLEG): container finished" podID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerID="c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed" exitCode=0
Oct 30 12:36:56.741603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.741408    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"
Oct 30 12:36:56.741603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.741436    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" event={"ID":"e8b2bceb-8b85-4d64-a4ee-03a590df1fed","Type":"ContainerDied","Data":"c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed"}
Oct 30 12:36:56.741603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.741526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2" event={"ID":"e8b2bceb-8b85-4d64-a4ee-03a590df1fed","Type":"ContainerDied","Data":"36178b80783ef98ee1cb7d3da87af9d11aabd3ff2cb1c581c0d1231fcdd63806"}
Oct 30 12:36:56.741603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.741562    5060 scope.go:122] "RemoveContainer" containerID="c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed"
Oct 30 12:36:56.753675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.753658    5060 scope.go:122] "RemoveContainer" containerID="c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed"
Oct 30 12:36:56.753961 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:36:56.753942    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed\": container with ID starting with c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed not found: ID does not exist" containerID="c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed"
Oct 30 12:36:56.754018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.753969    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed"} err="failed to get container status \"c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed\": rpc error: code = NotFound desc = could not find container \"c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed\": container with ID starting with c433e001a670b458c9e44113d16bc609debb34104d7cc7813d212ead1fae78ed not found: ID does not exist"
Oct 30 12:36:56.795657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.795623    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"]
Oct 30 12:36:56.810412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:56.810373    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd0ff7-78b88c6c79-k89s2"]
Oct 30 12:36:57.204518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:57.204434    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:58.335848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:58.335809    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" path="/var/lib/kubelet/pods/e8b2bceb-8b85-4d64-a4ee-03a590df1fed/volumes"
Oct 30 12:36:58.967996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:58.967438    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:59.734853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:59.734812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:59.876159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:36:59.876119    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:00.698416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.698380    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"]
Oct 30 12:37:00.699622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.699589    5060 cpu_manager.go:414] "containerMap: removing container" podUID="946659b4-7354-4e88-8e7b-13b8899bc0ed" containerName="revoke-keys"
Oct 30 12:37:00.699622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.699621    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946659b4-7354-4e88-8e7b-13b8899bc0ed" containerName="revoke-keys"
Oct 30 12:37:00.699892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.699645    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerName="payload-pre-processing"
Oct 30 12:37:00.699892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.699653    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerName="payload-pre-processing"
Oct 30 12:37:00.699892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.699873    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="946659b4-7354-4e88-8e7b-13b8899bc0ed" containerName="revoke-keys"
Oct 30 12:37:00.699892 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.699891    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2bceb-8b85-4d64-a4ee-03a590df1fed" containerName="payload-pre-processing"
Oct 30 12:37:00.705651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.705624    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.722422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.722386    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"]
Oct 30 12:37:00.736481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.728670    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"openshift-service-ca.crt\""
Oct 30 12:37:00.736481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.728682    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"kube-root-ca.crt\""
Oct 30 12:37:00.736481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.732726    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"default-dockercfg-l9bt7\""
Oct 30 12:37:00.736481 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.732851    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4fff4f\"/\"sub-test-model-4fff4f-kserve-self-signed-certs\""
Oct 30 12:37:00.757746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.757701    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:00.761672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.761648    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hltt\" (UniqueName: \"kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.761810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.761709    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.761961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.761933    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.762020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.761992    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.762121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.762099    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.762246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.762138    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863058    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863115    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863153    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863180    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863242    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hltt\" (UniqueName: \"kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863331    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863590    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863607    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.863991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.863959    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.865827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.865796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.866143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.866126    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.888115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.888084    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hltt\" (UniqueName: \"kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt\") pod \"sub-test-model-4fff4f-kserve-769df8f78f-hkmxd\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") " pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:00.902793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:00.902741    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:01.036529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:01.035901    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:01.439514 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:37:01.439483    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c8db401_80b6_4da8_8a88_1e183e80b2be.slice/crio-9662ca418277d2c73b3d0f7371f29740d15b5e02eee0c641f2246564b910c2f3 WatchSource:0}: Error finding container 9662ca418277d2c73b3d0f7371f29740d15b5e02eee0c641f2246564b910c2f3: Status 404 returned error can't find the container with id 9662ca418277d2c73b3d0f7371f29740d15b5e02eee0c641f2246564b910c2f3
Oct 30 12:37:01.441953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:01.441927    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"]
Oct 30 12:37:01.793589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:01.792015    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd" event={"ID":"2c8db401-80b6-4da8-8a88-1e183e80b2be","Type":"ContainerStarted","Data":"10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c"}
Oct 30 12:37:01.793589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:01.792086    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd" event={"ID":"2c8db401-80b6-4da8-8a88-1e183e80b2be","Type":"ContainerStarted","Data":"9662ca418277d2c73b3d0f7371f29740d15b5e02eee0c641f2246564b910c2f3"}
Oct 30 12:37:01.838537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:01.838470    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd" podStartSLOduration=1.838443507 podStartE2EDuration="1.838443507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:01.834129019 +0000 UTC m=+4025.629013512" watchObservedRunningTime="2026-10-30 12:37:01.838443507 +0000 UTC m=+4025.633328054"
Oct 30 12:37:02.037244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:02.037211    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:02.516381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:02.516341    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:03.224389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:03.224346    5060 scope.go:122] "RemoveContainer" containerID="83538fd78290011dd5d84be6d25ac4cf00cb0d2015a75f32e4eb27544378ee97"
Oct 30 12:37:03.236207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:03.236183    5060 scope.go:122] "RemoveContainer" containerID="1fa120590fd0b5527de5a665ea7fc32cb7e5926234518a115fce804c800ee362"
Oct 30 12:37:03.249443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:03.249418    5060 scope.go:122] "RemoveContainer" containerID="d395aeac6f880111b17ffe87d282c816bfa2d6a198f6a15636c878b3222beb10"
Oct 30 12:37:03.259788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:03.259766    5060 scope.go:122] "RemoveContainer" containerID="176125918f676537bb681b30756aa60cf992b79f02ef618d11194e85ce6662e4"
Oct 30 12:37:04.244310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.244274    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"]
Oct 30 12:37:04.250005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.249981    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.266551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.266524    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"]
Oct 30 12:37:04.268509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.268327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:04.272781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.272755    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:37:04.282033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.278439    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:37:04.282033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.278695    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:37:04.282506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.282299    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:37:04.319813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.319772    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.320029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.319904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqg9\" (UniqueName: \"kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.320029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.319988    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.421162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.421129    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.421366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.421183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqg9\" (UniqueName: \"kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.421366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.421222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.422013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.421990    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.423750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.423726    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.444673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.444646    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqg9\" (UniqueName: \"kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9\") pod \"maas-api-revoke-sub-a403d2d8-xdmh7\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.569953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.569870    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:04.877887 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:37:04.877856    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab40780_0645_40c8_a142_ad820778924f.slice/crio-6f923ec9502e0bb99487075c7aa5a7eeee01236a5edecf618b51e5423a024c46 WatchSource:0}: Error finding container 6f923ec9502e0bb99487075c7aa5a7eeee01236a5edecf618b51e5423a024c46: Status 404 returned error can't find the container with id 6f923ec9502e0bb99487075c7aa5a7eeee01236a5edecf618b51e5423a024c46
Oct 30 12:37:04.882088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:04.882057    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"]
Oct 30 12:37:05.831939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:05.831906    5060 generic.go:356] "Generic (PLEG): container finished" podID="bab40780-0645-40c8-a142-ad820778924f" containerID="4bad47c9baaaed5aca240f0003f7d63db485ca009272a0d8e202c29e5d96288f" exitCode=0
Oct 30 12:37:05.832377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:05.832023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7" event={"ID":"bab40780-0645-40c8-a142-ad820778924f","Type":"ContainerDied","Data":"4bad47c9baaaed5aca240f0003f7d63db485ca009272a0d8e202c29e5d96288f"}
Oct 30 12:37:05.832377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:05.832083    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7" event={"ID":"bab40780-0645-40c8-a142-ad820778924f","Type":"ContainerStarted","Data":"6f923ec9502e0bb99487075c7aa5a7eeee01236a5edecf618b51e5423a024c46"}
Oct 30 12:37:06.062119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:06.062035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:07.026854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.026819    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:07.060383 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.060343    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca\") pod \"bab40780-0645-40c8-a142-ad820778924f\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") "
Oct 30 12:37:07.060628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.060597    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9\" (UniqueName: \"kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9\") pod \"bab40780-0645-40c8-a142-ad820778924f\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") "
Oct 30 12:37:07.060713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.060663    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca\") pod \"bab40780-0645-40c8-a142-ad820778924f\" (UID: \"bab40780-0645-40c8-a142-ad820778924f\") "
Oct 30 12:37:07.060781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.060703    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca" pod "bab40780-0645-40c8-a142-ad820778924f" (UID: "bab40780-0645-40c8-a142-ad820778924f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:07.061157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.061138    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab40780-0645-40c8-a142-ad820778924f-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.062867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.062843    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca" pod "bab40780-0645-40c8-a142-ad820778924f" (UID: "bab40780-0645-40c8-a142-ad820778924f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:07.062994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.062973    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9" pod "bab40780-0645-40c8-a142-ad820778924f" (UID: "bab40780-0645-40c8-a142-ad820778924f"). InnerVolumeSpecName "kube-api-access-wbqg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:07.162747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.162666    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab40780-0645-40c8-a142-ad820778924f-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.162747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.162698    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbqg9\" (UniqueName: \"kubernetes.io/projected/bab40780-0645-40c8-a142-ad820778924f-kube-api-access-wbqg9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.300886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.300846    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:07.459182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.459126    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:07.676100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.676055    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:07.866523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.866417    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7" event={"ID":"bab40780-0645-40c8-a142-ad820778924f","Type":"ContainerDied","Data":"6f923ec9502e0bb99487075c7aa5a7eeee01236a5edecf618b51e5423a024c46"}
Oct 30 12:37:07.866773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.866757    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f923ec9502e0bb99487075c7aa5a7eeee01236a5edecf618b51e5423a024c46"
Oct 30 12:37:07.866933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:07.866922    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"
Oct 30 12:37:08.208960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.208923    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:08.321005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.320969    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"]
Oct 30 12:37:08.325256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.325233    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd0ff7-nhr8q"]
Oct 30 12:37:08.337603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.337570    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8ecb68-0066-42cc-adbf-77598d0eacc3" path="/var/lib/kubelet/pods/ae8ecb68-0066-42cc-adbf-77598d0eacc3/volumes"
Oct 30 12:37:08.354989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.354952    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:08.505523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.505413    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:08.843497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:08.843381    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:10.179617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:10.179579    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:11.037243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:11.037203    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:11.055910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:11.055873    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:11.855283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:11.855246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:11.922377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:11.922343    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:37:13.651907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:13.650482    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:14.709596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:14.709556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:15.433820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:15.433783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:15.783048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:15.783013    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:15.930376 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:15.930345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:16.101937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:16.099958    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"]
Oct 30 12:37:16.101937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:16.100226    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy" containerID="cri-o://a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1" gracePeriod=30
Oct 30 12:37:16.167817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:16.167778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:17.116389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:17.116351    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:17.280950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:17.280915    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:17.529981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:17.529949    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:19.136188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:19.136133    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:19.858543 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:37:19.858507    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:37:20.890532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:20.890489    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:21.392597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.392470    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:21.408430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.408400    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:37:21.441244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441214    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441260    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441298    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441338    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441366    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441413    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441475    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441532    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token\") pod \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\" (UID: \"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a\") "
Oct 30 12:37:21.441784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441584    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:21.441784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441730    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:21.441784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441749    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:21.442152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441810    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:21.442152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.441947    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:21.442615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.442591    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.442718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.442618    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.442718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.442635    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.442718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.442651    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.442718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.442664    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.443925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.443901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:21.444040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.443954    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:37:21.444088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.444073    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:21.444129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.444094    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs" pod "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" (UID: "6a987869-2c8a-4ac3-85e1-2f3ebceaf85a"). InnerVolumeSpecName "kube-api-access-bwxqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:21.544189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.544112    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.544189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.544138    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.544189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.544148    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxqs\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-kube-api-access-bwxqs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:21.544189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:21.544166    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:22.004284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.004246    5060 generic.go:356] "Generic (PLEG): container finished" podID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerID="a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1" exitCode=0
Oct 30 12:37:22.004802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.004350    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" event={"ID":"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a","Type":"ContainerDied","Data":"a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1"}
Oct 30 12:37:22.004802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.004388    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb" event={"ID":"6a987869-2c8a-4ac3-85e1-2f3ebceaf85a","Type":"ContainerDied","Data":"c9c7e6aac6800792e06e6cd36218c3dc233f89d9422b9ef240b0bbc7bc439f6b"}
Oct 30 12:37:22.004802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.004432    5060 scope.go:122] "RemoveContainer" containerID="a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1"
Oct 30 12:37:22.004802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.004584    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"
Oct 30 12:37:22.020466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.020428    5060 scope.go:122] "RemoveContainer" containerID="ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"
Oct 30 12:37:22.031648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.031628    5060 scope.go:122] "RemoveContainer" containerID="a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1"
Oct 30 12:37:22.031952 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:37:22.031933    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1\": container with ID starting with a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1 not found: ID does not exist" containerID="a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1"
Oct 30 12:37:22.032008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.031961    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1"} err="failed to get container status \"a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1\": rpc error: code = NotFound desc = could not find container \"a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1\": container with ID starting with a917719acd7893fc900c7c2d0feb52d142d1ac69cbfbd4ac995962c254c1d8c1 not found: ID does not exist"
Oct 30 12:37:22.032008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.031978    5060 scope.go:122] "RemoveContainer" containerID="ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"
Oct 30 12:37:22.032217 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:37:22.032194    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5\": container with ID starting with ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5 not found: ID does not exist" containerID="ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"
Oct 30 12:37:22.032330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.032218    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5"} err="failed to get container status \"ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5\": rpc error: code = NotFound desc = could not find container \"ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5\": container with ID starting with ca0aa7ebaa49c97b2615a1a6f906ff320b44b6aa30d08951234be3e88d5150b5 not found: ID does not exist"
Oct 30 12:37:22.060982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.060945    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"]
Oct 30 12:37:22.073058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.073012    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd0ff7-openshift-default-7f4f68fc6c-4xgnb"]
Oct 30 12:37:22.335881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.335805    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" path="/var/lib/kubelet/pods/6a987869-2c8a-4ac3-85e1-2f3ebceaf85a/volumes"
Oct 30 12:37:22.668729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:22.668629    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:23.825096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:23.825053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:23.979480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:23.979388    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:24.115505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.115404    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"]
Oct 30 12:37:24.116140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116124    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bab40780-0645-40c8-a142-ad820778924f" containerName="revoke-keys"
Oct 30 12:37:24.116193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116142    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab40780-0645-40c8-a142-ad820778924f" containerName="revoke-keys"
Oct 30 12:37:24.116193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116170    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy"
Oct 30 12:37:24.116193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116176    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy"
Oct 30 12:37:24.116349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116302    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy"
Oct 30 12:37:24.116349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116317    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy"
Oct 30 12:37:24.116349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.116327    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab40780-0645-40c8-a142-ad820778924f" containerName="revoke-keys"
Oct 30 12:37:24.121618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.121597    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.137115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.137087    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:37:24.139821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.139793    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"]
Oct 30 12:37:24.146257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.146226    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:37:24.186937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.186900    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:37:24.187135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.186954    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:37:24.187188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.187152    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt767\" (UniqueName: \"kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.187251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.187201    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.187396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.187379    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.289408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.289372    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.289669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.289542    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt767\" (UniqueName: \"kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.289669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.289595    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.290285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.290261    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.292081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.292056    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.314016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.313976    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt767\" (UniqueName: \"kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767\") pod \"maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.441082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.440992    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:24.571759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.571720    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:24.813697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:24.811986    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"]
Oct 30 12:37:24.820281 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:37:24.820254    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ce1d1c_382b_4ca1_b30e_a9530dc7dfb1.slice/crio-b63cc76d17d64836358a0e1ecec5cfb998ed806f484481602d6350f5b741a4a5 WatchSource:0}: Error finding container b63cc76d17d64836358a0e1ecec5cfb998ed806f484481602d6350f5b741a4a5: Status 404 returned error can't find the container with id b63cc76d17d64836358a0e1ecec5cfb998ed806f484481602d6350f5b741a4a5
Oct 30 12:37:25.048977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:25.048940    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv" event={"ID":"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1","Type":"ContainerStarted","Data":"b63cc76d17d64836358a0e1ecec5cfb998ed806f484481602d6350f5b741a4a5"}
Oct 30 12:37:26.059738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:26.059698    5060 generic.go:356] "Generic (PLEG): container finished" podID="f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" containerID="58acc6e1a01be75e8bc095b028d86f6aff1a1d64c084e95cad421a8961c37546" exitCode=0
Oct 30 12:37:26.060170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:26.059784    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv" event={"ID":"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1","Type":"ContainerDied","Data":"58acc6e1a01be75e8bc095b028d86f6aff1a1d64c084e95cad421a8961c37546"}
Oct 30 12:37:26.505474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:26.503413    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:27.247965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.247940    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:27.375055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.374973    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767\" (UniqueName: \"kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767\") pod \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") "
Oct 30 12:37:27.375055 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.375016    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca\") pod \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") "
Oct 30 12:37:27.375282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.375190    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca\") pod \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\" (UID: \"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1\") "
Oct 30 12:37:27.375597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.375572    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca" pod "f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" (UID: "f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:27.377275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.377235    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767" pod "f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" (UID: "f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1"). InnerVolumeSpecName "kube-api-access-qt767". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:27.377667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.377649    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca" pod "f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" (UID: "f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:27.477930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.477887    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:27.478122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.477996    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt767\" (UniqueName: \"kubernetes.io/projected/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-kube-api-access-qt767\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:27.478122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:27.478022    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:28.081139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:28.081098    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"
Oct 30 12:37:28.081373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:28.081133    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv" event={"ID":"f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1","Type":"ContainerDied","Data":"b63cc76d17d64836358a0e1ecec5cfb998ed806f484481602d6350f5b741a4a5"}
Oct 30 12:37:28.081373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:28.081203    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b63cc76d17d64836358a0e1ecec5cfb998ed806f484481602d6350f5b741a4a5"
Oct 30 12:37:28.476678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:28.476547    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:30.241242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:30.241207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:31.705000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:31.704946    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:32.449347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:32.449308    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:32.563638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:32.563600    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"]
Oct 30 12:37:32.579606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:32.579574    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffea51-t2tt7"]
Oct 30 12:37:32.592569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:32.592528    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:32.738333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:32.738243    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:33.109774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:33.109681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:34.335745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:34.335712    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946659b4-7354-4e88-8e7b-13b8899bc0ed" path="/var/lib/kubelet/pods/946659b4-7354-4e88-8e7b-13b8899bc0ed/volumes"
Oct 30 12:37:34.666275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:34.666187    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:34.839549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:34.839514    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:36.671185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:36.671151    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:38.614191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:38.614126    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:39.293933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:39.293894    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:39.439331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:39.439290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:39.533000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:39.532967    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"]
Oct 30 12:37:39.533246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:39.533214    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy" containerID="cri-o://863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c" gracePeriod=30
Oct 30 12:37:39.591335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:39.591238    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:40.401643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:40.401603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:40.559362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:40.559323    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:40.887169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:40.887133    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:41.043616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:41.043580    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:41.876260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:41.876225    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:42.362866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:42.362832    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:42.782595 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:42.782553    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:42.928040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:42.928000    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:43.472421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:43.472382    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:44.334511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:44.334443    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:44.786818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:44.786769    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:44.923471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:44.923424    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:45.230515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.230489    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:37:45.253601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.253561    5060 generic.go:356] "Generic (PLEG): container finished" podID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerID="863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c" exitCode=0
Oct 30 12:37:45.253772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.253632    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"
Oct 30 12:37:45.253772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.253659    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" event={"ID":"68bf4d13-9763-4407-9cd5-0fa4bbb628b1","Type":"ContainerDied","Data":"863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c"}
Oct 30 12:37:45.253772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.253721    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl" event={"ID":"68bf4d13-9763-4407-9cd5-0fa4bbb628b1","Type":"ContainerDied","Data":"29338aa355ffe5ba9cd6efc77984f8a11d9e6ac1a280d4892b73b9f972f76644"}
Oct 30 12:37:45.253772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.253753    5060 scope.go:122] "RemoveContainer" containerID="863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c"
Oct 30 12:37:45.268716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.268693    5060 scope.go:122] "RemoveContainer" containerID="b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"
Oct 30 12:37:45.281345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.281323    5060 scope.go:122] "RemoveContainer" containerID="863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c"
Oct 30 12:37:45.281695 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:37:45.281673    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c\": container with ID starting with 863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c not found: ID does not exist" containerID="863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c"
Oct 30 12:37:45.281771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.281706    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c"} err="failed to get container status \"863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c\": rpc error: code = NotFound desc = could not find container \"863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c\": container with ID starting with 863fd63db59d39a4b69540b23d4afda5da344d5778cbf82cceb5b9d4d725449c not found: ID does not exist"
Oct 30 12:37:45.281771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.281724    5060 scope.go:122] "RemoveContainer" containerID="b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"
Oct 30 12:37:45.282019 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:37:45.282003    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085\": container with ID starting with b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085 not found: ID does not exist" containerID="b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"
Oct 30 12:37:45.282065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.282022    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085"} err="failed to get container status \"b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085\": rpc error: code = NotFound desc = could not find container \"b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085\": container with ID starting with b71b0ec710b1f8864eb132965e3b4152bb0e28089ba4d7179f7280b8288d4085 not found: ID does not exist"
Oct 30 12:37:45.344140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344042    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344203    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344241    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344308    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344417    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344437    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344513    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344526    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:45.344670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket\") pod \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\" (UID: \"68bf4d13-9763-4407-9cd5-0fa4bbb628b1\") "
Oct 30 12:37:45.345208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.344815    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:45.345208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345040    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:45.345208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345069    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:45.345400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345280    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.345400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:45.345400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345295    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.345400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345329    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.345400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.345346    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.347498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.347403    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:45.347498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.347442    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:37:45.347809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.347783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:45.349419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.349387    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7" pod "68bf4d13-9763-4407-9cd5-0fa4bbb628b1" (UID: "68bf4d13-9763-4407-9cd5-0fa4bbb628b1"). InnerVolumeSpecName "kube-api-access-h4vh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:45.447408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.447368    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.447408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.447392    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.447408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.447402    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vh7\" (UniqueName: \"kubernetes.io/projected/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-kube-api-access-h4vh7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.447408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.447413    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.447408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.447422    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bf4d13-9763-4407-9cd5-0fa4bbb628b1-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:37:45.603853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.603765    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"]
Oct 30 12:37:45.615642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:45.615606    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ffea51-openshift-default-67ff9c8fbf-kh5nl"]
Oct 30 12:37:46.274730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:46.274686    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:46.346388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:46.346353    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" path="/var/lib/kubelet/pods/68bf4d13-9763-4407-9cd5-0fa4bbb628b1/volumes"
Oct 30 12:37:46.513888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:46.513848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:46.665656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:46.665565    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:48.208991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:48.208954    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:49.860474 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:37:49.860421    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:37:49.906765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:49.906727    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:50.046299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:50.046260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:51.757815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:51.757779    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:53.438138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:53.438100    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:55.151495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:55.150640    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:56.271868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:56.271828    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:56.855578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:56.855525    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:58.598532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:37:58.598496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:00.387241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:00.383405    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:02.782840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:02.782792    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:03.040589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:03.040499    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"]
Oct 30 12:38:03.052745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:03.052713    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1d70d0f-8n9gd"]
Oct 30 12:38:03.334177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:03.334077    5060 scope.go:122] "RemoveContainer" containerID="79581e6d1ed60e3333bc2d9d3e03be45ec44bb0200744605d2e76755a5daee56"
Oct 30 12:38:04.336251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:04.336206    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c157d9-e3cb-4b87-b145-268bfca0a860" path="/var/lib/kubelet/pods/56c157d9-e3cb-4b87-b145-268bfca0a860/volumes"
Oct 30 12:38:04.440467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:04.440414    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:06.154884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:06.154829    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:07.948043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:07.948005    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:09.621836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:09.621791    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:11.472826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:11.472788    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:12.904998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:12.904954    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:13.620271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:13.620233    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:13.723878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:13.723842    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"]
Oct 30 12:38:13.735085 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:13.735059    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-59b8a9-lwxsv"]
Oct 30 12:38:13.765238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:13.765205    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:14.264081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:14.264047    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:14.337183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:14.337152    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" path="/var/lib/kubelet/pods/f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1/volumes"
Oct 30 12:38:15.031086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:15.031049    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"]
Oct 30 12:38:15.043555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:15.043522    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba98891-cg5sm"]
Oct 30 12:38:15.536142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:15.536106    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:16.339196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:16.339157    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de5723b-36f4-40cb-946c-99b3f1939305" path="/var/lib/kubelet/pods/9de5723b-36f4-40cb-946c-99b3f1939305/volumes"
Oct 30 12:38:17.358818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:17.358784    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:19.055643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:19.055584    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:19.857441 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:38:19.857408    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:38:20.905868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:20.905799    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:21.046270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.046237    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"]
Oct 30 12:38:21.046524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.046497    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" containerID="cri-o://331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d" gracePeriod=30
Oct 30 12:38:21.060724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.060505    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:21.204722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.204685    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:21.641277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641183    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"]
Oct 30 12:38:21.641915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641897    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" containerName="revoke-keys"
Oct 30 12:38:21.641998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641916    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" containerName="revoke-keys"
Oct 30 12:38:21.641998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641925    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy"
Oct 30 12:38:21.641998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641930    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy"
Oct 30 12:38:21.641998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641952    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy"
Oct 30 12:38:21.641998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.641957    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a987869-2c8a-4ac3-85e1-2f3ebceaf85a" containerName="istio-proxy"
Oct 30 12:38:21.642176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.642075    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy"
Oct 30 12:38:21.642176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.642090    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ce1d1c-382b-4ca1-b30e-a9530dc7dfb1" containerName="revoke-keys"
Oct 30 12:38:21.642176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.642096    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy"
Oct 30 12:38:21.647447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.647428    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.662561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.662535    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"]
Oct 30 12:38:21.670281 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.670256    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:38:21.670432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.670289    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:38:21.674500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.674477    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:38:21.674763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.674741    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:38:21.745794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.745754    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27p6k\" (UniqueName: \"kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.745977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.745832    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.746027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.745996    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.846975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.846937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.847160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.847134    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.847267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.847251    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27p6k\" (UniqueName: \"kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.847852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.847768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.849542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.849518    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.873957 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.873933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27p6k\" (UniqueName: \"kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k\") pod \"maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:21.963262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:21.963237    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:22.273811 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:38:22.273765    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d978ed_3c6f_40c1_bc40_58a30c53c8a9.slice/crio-25ba2e41804b5e8512461637d000fe6a8531b2522cc4d6433575760d2836e7bf WatchSource:0}: Error finding container 25ba2e41804b5e8512461637d000fe6a8531b2522cc4d6433575760d2836e7bf: Status 404 returned error can't find the container with id 25ba2e41804b5e8512461637d000fe6a8531b2522cc4d6433575760d2836e7bf
Oct 30 12:38:22.275993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:22.275969    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"]
Oct 30 12:38:22.561065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:22.560984    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:22.651477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:22.651429    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh" event={"ID":"22d978ed-3c6f-40c1-bc40-58a30c53c8a9","Type":"ContainerStarted","Data":"25ba2e41804b5e8512461637d000fe6a8531b2522cc4d6433575760d2836e7bf"}
Oct 30 12:38:23.002522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:23.002482    5060 patch_prober.go:29] interesting pod/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:38:23.002943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:23.002554    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:38:23.661156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:23.661046    5060 generic.go:356] "Generic (PLEG): container finished" podID="22d978ed-3c6f-40c1-bc40-58a30c53c8a9" containerID="35b1d50f2a2d388d4d17885bf5b9c7e39f00e17748c4787383bd06baa03d8724" exitCode=0
Oct 30 12:38:23.661156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:23.661138    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh" event={"ID":"22d978ed-3c6f-40c1-bc40-58a30c53c8a9","Type":"ContainerDied","Data":"35b1d50f2a2d388d4d17885bf5b9c7e39f00e17748c4787383bd06baa03d8724"}
Oct 30 12:38:24.244212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.244168    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:24.856167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.856140    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:24.901086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.901058    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca\") pod \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") "
Oct 30 12:38:24.901254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.901135    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca\") pod \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") "
Oct 30 12:38:24.901324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.901273    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k\" (UniqueName: \"kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k\") pod \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\" (UID: \"22d978ed-3c6f-40c1-bc40-58a30c53c8a9\") "
Oct 30 12:38:24.902510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.902441    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca" pod "22d978ed-3c6f-40c1-bc40-58a30c53c8a9" (UID: "22d978ed-3c6f-40c1-bc40-58a30c53c8a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:38:24.904365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.904332    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k" pod "22d978ed-3c6f-40c1-bc40-58a30c53c8a9" (UID: "22d978ed-3c6f-40c1-bc40-58a30c53c8a9"). InnerVolumeSpecName "kube-api-access-27p6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:38:24.916700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:24.916627    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca" pod "22d978ed-3c6f-40c1-bc40-58a30c53c8a9" (UID: "22d978ed-3c6f-40c1-bc40-58a30c53c8a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:38:25.003143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.003108    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:25.003143 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.003138    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27p6k\" (UniqueName: \"kubernetes.io/projected/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-kube-api-access-27p6k\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:25.003328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.003151    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d978ed-3c6f-40c1-bc40-58a30c53c8a9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:25.679787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.679754    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh" event={"ID":"22d978ed-3c6f-40c1-bc40-58a30c53c8a9","Type":"ContainerDied","Data":"25ba2e41804b5e8512461637d000fe6a8531b2522cc4d6433575760d2836e7bf"}
Oct 30 12:38:25.679787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.679777    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"
Oct 30 12:38:25.680242 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.679801    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ba2e41804b5e8512461637d000fe6a8531b2522cc4d6433575760d2836e7bf"
Oct 30 12:38:25.923599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:25.923560    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:26.033433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.033393    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"]
Oct 30 12:38:26.037329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.037302    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d7505b-g5lbk"]
Oct 30 12:38:26.338502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.338477    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732cfea0-b4ca-43a6-9be1-b8f6a6a51be3" path="/var/lib/kubelet/pods/732cfea0-b4ca-43a6-9be1-b8f6a6a51be3/volumes"
Oct 30 12:38:26.339694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.339672    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:38:26.421844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.421805    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.421869    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.421931    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.421992    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.422025    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.422117    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.422164    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.422209    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.422328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.422280    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo\") pod \"85e29aab-e875-464e-b2e7-3cfb50e236a8\" (UID: \"85e29aab-e875-464e-b2e7-3cfb50e236a8\") "
Oct 30 12:38:26.423263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.423234    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:38:26.423510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.423441    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:38:26.423997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.423978    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:38:26.424441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.424414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:38:26.424779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.424757    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:38:26.425855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.425819    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "kube-api-access-csfch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:38:26.427729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.427706    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:38:26.427837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.427814    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:38:26.428496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.428468    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token" pod "85e29aab-e875-464e-b2e7-3cfb50e236a8" (UID: "85e29aab-e875-464e-b2e7-3cfb50e236a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:38:26.524201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524165    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfch\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-kube-api-access-csfch\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524202    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524216    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524231    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e29aab-e875-464e-b2e7-3cfb50e236a8-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524243    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524255    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524268    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524279    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e29aab-e875-464e-b2e7-3cfb50e236a8-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.524428 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.524293    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e29aab-e875-464e-b2e7-3cfb50e236a8-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:38:26.694386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.694285    5060 generic.go:356] "Generic (PLEG): container finished" podID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerID="331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d" exitCode=0
Oct 30 12:38:26.694386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.694343    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"
Oct 30 12:38:26.694386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.694371    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" event={"ID":"85e29aab-e875-464e-b2e7-3cfb50e236a8","Type":"ContainerDied","Data":"331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d"}
Oct 30 12:38:26.695000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.694429    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl" event={"ID":"85e29aab-e875-464e-b2e7-3cfb50e236a8","Type":"ContainerDied","Data":"bcf2d7adddf29311ef5273b73cb6a4bcd87254c2a8dac56b160a78605c0c1df1"}
Oct 30 12:38:26.695000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.694483    5060 scope.go:122] "RemoveContainer" containerID="331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d"
Oct 30 12:38:26.708126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.708109    5060 scope.go:122] "RemoveContainer" containerID="0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"
Oct 30 12:38:26.719189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.719169    5060 scope.go:122] "RemoveContainer" containerID="331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d"
Oct 30 12:38:26.719441 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:38:26.719424    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d\": container with ID starting with 331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d not found: ID does not exist" containerID="331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d"
Oct 30 12:38:26.719651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.719480    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d"} err="failed to get container status \"331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d\": rpc error: code = NotFound desc = could not find container \"331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d\": container with ID starting with 331c5cb02e3a66e53b6ff1a0074d29e52fe70147c1b0c75d04bfd79f7215889d not found: ID does not exist"
Oct 30 12:38:26.719651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.719503    5060 scope.go:122] "RemoveContainer" containerID="0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"
Oct 30 12:38:26.719784 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:38:26.719752    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260\": container with ID starting with 0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260 not found: ID does not exist" containerID="0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"
Oct 30 12:38:26.719784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.719775    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260"} err="failed to get container status \"0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260\": rpc error: code = NotFound desc = could not find container \"0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260\": container with ID starting with 0719cac774705f39131c5dce7d9cdfd1b39b7f1229b9a16cf71f3f0b9dc97260 not found: ID does not exist"
Oct 30 12:38:26.751906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.751879    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"]
Oct 30 12:38:26.762987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:26.762964    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-59b8a9-openshift-default-cb4954c7c-qm9cl"]
Oct 30 12:38:27.723889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:27.723855    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:27.895530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:27.895494    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:28.036335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:28.036251    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:28.336074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:28.335996    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" path="/var/lib/kubelet/pods/85e29aab-e875-464e-b2e7-3cfb50e236a8/volumes"
Oct 30 12:38:29.546493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:29.546425    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:31.216944 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:31.216905    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:32.339446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:32.339416    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:32.494187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:32.494143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:33.066966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:33.066918    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:34.874569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:34.874533    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:36.638231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:36.638193    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:38.404010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:38.403969    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:40.020358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:40.020323    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:41.763586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:41.763553    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:43.544912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:43.544877    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:45.336201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:45.336170    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:47.073377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:47.073340    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:48.758013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:48.757976    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:49.857370 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:38:49.857331    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:38:50.643378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:50.643327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:52.300271 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:52.300238    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:54.079014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:54.078974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:55.949053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:55.949012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:56.402500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:56.402372    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:57.756000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:57.755966    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:59.492171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:38:59.492121    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:01.325961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:01.325923    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:03.052717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:03.052677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:03.380190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:03.380101    5060 scope.go:122] "RemoveContainer" containerID="15651cdd2d8bf970e88f4a0249250f18cb0e911a92eb19d4b47191e78d587605"
Oct 30 12:39:04.568396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:04.568358    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:05.116121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:05.116058    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:05.211308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:05.211259    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"]
Oct 30 12:39:05.223135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:05.223100    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b15ec3-xmjdh"]
Oct 30 12:39:05.251069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:05.251035    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:05.394180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:05.394100    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:05.742855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:05.742817    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:06.338604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:06.338572    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d978ed-3c6f-40c1-bc40-58a30c53c8a9" path="/var/lib/kubelet/pods/22d978ed-3c6f-40c1-bc40-58a30c53c8a9/volumes"
Oct 30 12:39:07.250863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:07.250826    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:09.057240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:09.057199    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:10.882947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:10.882907    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:12.593796 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:12.593743    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:12.736372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:12.736335    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:12.874633 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:12.874535    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"]
Oct 30 12:39:12.874830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:12.874785    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy" containerID="cri-o://e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762" gracePeriod=30
Oct 30 12:39:12.884139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:12.884105    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:13.039293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.039257    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:13.452359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.452323    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"]
Oct 30 12:39:13.453002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.452987    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy"
Oct 30 12:39:13.453073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453003    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bf4d13-9763-4407-9cd5-0fa4bbb628b1" containerName="istio-proxy"
Oct 30 12:39:13.453073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453037    5060 cpu_manager.go:414] "containerMap: removing container" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy"
Oct 30 12:39:13.453073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453043    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy"
Oct 30 12:39:13.453073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453064    5060 cpu_manager.go:414] "containerMap: removing container" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy"
Oct 30 12:39:13.453073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453069    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy"
Oct 30 12:39:13.453251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453077    5060 cpu_manager.go:414] "containerMap: removing container" podUID="22d978ed-3c6f-40c1-bc40-58a30c53c8a9" containerName="revoke-keys"
Oct 30 12:39:13.453251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453082    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d978ed-3c6f-40c1-bc40-58a30c53c8a9" containerName="revoke-keys"
Oct 30 12:39:13.453251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453172    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy"
Oct 30 12:39:13.453251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.453184    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d978ed-3c6f-40c1-bc40-58a30c53c8a9" containerName="revoke-keys"
Oct 30 12:39:13.465916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.465877    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.476717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.476684    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"]
Oct 30 12:39:13.484089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.484054    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:39:13.484236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.484139    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:39:13.525111 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.525080    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:39:13.525356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.525080    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:39:13.532213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.532187    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.532360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.532224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.532360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.532318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vj5\" (UniqueName: \"kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.634291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.634250    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.634744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.634299    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.634744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.634356    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vj5\" (UniqueName: \"kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.635014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.634992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.636987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.636963    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.659126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.659091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vj5\" (UniqueName: \"kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.783923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.783826    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:13.975813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.975768    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"]
Oct 30 12:39:13.976629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.976597    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e29aab-e875-464e-b2e7-3cfb50e236a8" containerName="istio-proxy"
Oct 30 12:39:13.980035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.980017    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:13.997724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:13.997694    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"]
Oct 30 12:39:14.039789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.039692    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.039789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.039748    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.040004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.039917    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k68ms\" (UniqueName: \"kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.097575 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:39:14.097547    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87f63ba_62ae_43cc_8abd_7afe63ca3f8c.slice/crio-be6f9cb0d00b37afabcdc817e09dc33ba9451715cb2eacf1453d56cd4eefb749 WatchSource:0}: Error finding container be6f9cb0d00b37afabcdc817e09dc33ba9451715cb2eacf1453d56cd4eefb749: Status 404 returned error can't find the container with id be6f9cb0d00b37afabcdc817e09dc33ba9451715cb2eacf1453d56cd4eefb749
Oct 30 12:39:14.099568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.099537    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"]
Oct 30 12:39:14.141309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.141281    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.141447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.141350    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.141542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.141496    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k68ms\" (UniqueName: \"kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.142010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.141991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.143707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.143688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.165149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.165123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k68ms\" (UniqueName: \"kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms\") pod \"maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.217007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.216972    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s" event={"ID":"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c","Type":"ContainerStarted","Data":"be6f9cb0d00b37afabcdc817e09dc33ba9451715cb2eacf1453d56cd4eefb749"}
Oct 30 12:39:14.296100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.296065    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:14.608762 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:39:14.608727    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216b72af_eff4_484e_8e8e_236829576c71.slice/crio-e40b11c718540823276445049892874dc2638baa6bb63f8d30ab7d313776b04e WatchSource:0}: Error finding container e40b11c718540823276445049892874dc2638baa6bb63f8d30ab7d313776b04e: Status 404 returned error can't find the container with id e40b11c718540823276445049892874dc2638baa6bb63f8d30ab7d313776b04e
Oct 30 12:39:14.610425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.610399    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"]
Oct 30 12:39:14.730310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:14.730264    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:15.229671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:15.229630    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5" event={"ID":"216b72af-eff4-484e-8e8e-236829576c71","Type":"ContainerStarted","Data":"7f6883062dcbef4ca0127a8a1ca5c290bfcd1108f9939274a5811ceb12456581"}
Oct 30 12:39:15.229887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:15.229700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5" event={"ID":"216b72af-eff4-484e-8e8e-236829576c71","Type":"ContainerStarted","Data":"e40b11c718540823276445049892874dc2638baa6bb63f8d30ab7d313776b04e"}
Oct 30 12:39:15.231684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:15.231655    5060 generic.go:356] "Generic (PLEG): container finished" podID="d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" containerID="77a61f344178bd04935a1cb2ba9a0254763348002289339a0f386c8cadd05de6" exitCode=0
Oct 30 12:39:15.231836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:15.231770    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s" event={"ID":"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c","Type":"ContainerDied","Data":"77a61f344178bd04935a1cb2ba9a0254763348002289339a0f386c8cadd05de6"}
Oct 30 12:39:15.273187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:15.273140    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5" podStartSLOduration=2.273126747 podStartE2EDuration="2.273126747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:15.269831076 +0000 UTC m=+4159.064715570" watchObservedRunningTime="2026-10-30 12:39:15.273126747 +0000 UTC m=+4159.068011241"
Oct 30 12:39:16.242275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.242241    5060 generic.go:356] "Generic (PLEG): container finished" podID="216b72af-eff4-484e-8e8e-236829576c71" containerID="7f6883062dcbef4ca0127a8a1ca5c290bfcd1108f9939274a5811ceb12456581" exitCode=0
Oct 30 12:39:16.242776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.242329    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5" event={"ID":"216b72af-eff4-484e-8e8e-236829576c71","Type":"ContainerDied","Data":"7f6883062dcbef4ca0127a8a1ca5c290bfcd1108f9939274a5811ceb12456581"}
Oct 30 12:39:16.422810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.422783    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:16.598216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.598119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca\") pod \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") "
Oct 30 12:39:16.598406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.598228    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca\") pod \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") "
Oct 30 12:39:16.598406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.598359    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5\" (UniqueName: \"kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5\") pod \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\" (UID: \"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c\") "
Oct 30 12:39:16.598654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.598623    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca" pod "d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" (UID: "d87f63ba-62ae-43cc-8abd-7afe63ca3f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:16.598894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.598878    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:16.600649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.600620    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca" pod "d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" (UID: "d87f63ba-62ae-43cc-8abd-7afe63ca3f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:39:16.600804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.600689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5" pod "d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" (UID: "d87f63ba-62ae-43cc-8abd-7afe63ca3f8c"). InnerVolumeSpecName "kube-api-access-s8vj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:16.615333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.615295    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:16.699999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.699962    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vj5\" (UniqueName: \"kubernetes.io/projected/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-kube-api-access-s8vj5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:16.699999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.699994    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:16.802941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:16.802901    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:17.251489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.251434    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"
Oct 30 12:39:17.251489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.251481    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s" event={"ID":"d87f63ba-62ae-43cc-8abd-7afe63ca3f8c","Type":"ContainerDied","Data":"be6f9cb0d00b37afabcdc817e09dc33ba9451715cb2eacf1453d56cd4eefb749"}
Oct 30 12:39:17.252044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.251555    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be6f9cb0d00b37afabcdc817e09dc33ba9451715cb2eacf1453d56cd4eefb749"
Oct 30 12:39:17.254040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.254011    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:17.417039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.417009    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:17.614841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.614742    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms\" (UniqueName: \"kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms\") pod \"216b72af-eff4-484e-8e8e-236829576c71\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") "
Oct 30 12:39:17.614841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.614799    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca\") pod \"216b72af-eff4-484e-8e8e-236829576c71\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") "
Oct 30 12:39:17.615098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.614912    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca\") pod \"216b72af-eff4-484e-8e8e-236829576c71\" (UID: \"216b72af-eff4-484e-8e8e-236829576c71\") "
Oct 30 12:39:17.615486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.615436    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca" pod "216b72af-eff4-484e-8e8e-236829576c71" (UID: "216b72af-eff4-484e-8e8e-236829576c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:17.618031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.617988    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms" pod "216b72af-eff4-484e-8e8e-236829576c71" (UID: "216b72af-eff4-484e-8e8e-236829576c71"). InnerVolumeSpecName "kube-api-access-k68ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:17.618170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.618087    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca" pod "216b72af-eff4-484e-8e8e-236829576c71" (UID: "216b72af-eff4-484e-8e8e-236829576c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:39:17.720691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.720610    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k68ms\" (UniqueName: \"kubernetes.io/projected/216b72af-eff4-484e-8e8e-236829576c71-kube-api-access-k68ms\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:17.720691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.720650    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b72af-eff4-484e-8e8e-236829576c71-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:17.720691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.720665    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b72af-eff4-484e-8e8e-236829576c71-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:17.759836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.759801    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:17.967591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:17.966188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:18.209078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.209054    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:39:18.254118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254073    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254158    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254199    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254313    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254342    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254487    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254558    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.254704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.254611    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket\") pod \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\" (UID: \"0c2f3122-ec0f-401f-b8b8-e37640160a8e\") "
Oct 30 12:39:18.255172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.255143    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.255399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.255324    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:18.255939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.255638    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:18.255939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.255890    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:18.256527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.256200    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:18.257870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.257846    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:39:18.258659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.258622    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "kube-api-access-98q22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:18.259652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.259618    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:18.259980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.259960    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy" pod "0c2f3122-ec0f-401f-b8b8-e37640160a8e" (UID: "0c2f3122-ec0f-401f-b8b8-e37640160a8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:18.268150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.268130    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"
Oct 30 12:39:18.268254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.268173    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5" event={"ID":"216b72af-eff4-484e-8e8e-236829576c71","Type":"ContainerDied","Data":"e40b11c718540823276445049892874dc2638baa6bb63f8d30ab7d313776b04e"}
Oct 30 12:39:18.268324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.268268    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e40b11c718540823276445049892874dc2638baa6bb63f8d30ab7d313776b04e"
Oct 30 12:39:18.271332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.271018    5060 generic.go:356] "Generic (PLEG): container finished" podID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerID="e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762" exitCode=0
Oct 30 12:39:18.271332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.271088    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"
Oct 30 12:39:18.271332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.271124    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" event={"ID":"0c2f3122-ec0f-401f-b8b8-e37640160a8e","Type":"ContainerDied","Data":"e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762"}
Oct 30 12:39:18.271332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.271163    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw" event={"ID":"0c2f3122-ec0f-401f-b8b8-e37640160a8e","Type":"ContainerDied","Data":"1d3be953223e7e80328093be5bc82874e5e3e7ead19291e053a42914b6585a82"}
Oct 30 12:39:18.271332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.271192    5060 scope.go:122] "RemoveContainer" containerID="e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762"
Oct 30 12:39:18.305400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.305380    5060 scope.go:122] "RemoveContainer" containerID="0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"
Oct 30 12:39:18.333899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.332670    5060 scope.go:122] "RemoveContainer" containerID="e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762"
Oct 30 12:39:18.333899 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:39:18.333654    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762\": container with ID starting with e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762 not found: ID does not exist" containerID="e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762"
Oct 30 12:39:18.333899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.333680    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762"} err="failed to get container status \"e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762\": rpc error: code = NotFound desc = could not find container \"e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762\": container with ID starting with e3208381ee5fa1bf98782fdc909278e49a143f53532c15b99098a096bb07d762 not found: ID does not exist"
Oct 30 12:39:18.333899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.333699    5060 scope.go:122] "RemoveContainer" containerID="0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"
Oct 30 12:39:18.334191 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:39:18.333975    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53\": container with ID starting with 0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53 not found: ID does not exist" containerID="0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"
Oct 30 12:39:18.334191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.334008    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53"} err="failed to get container status \"0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53\": rpc error: code = NotFound desc = could not find container \"0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53\": container with ID starting with 0c2eb453971f2035ca52206c8d389a1a6dca147aa9053fbc7285cf98913d7f53 not found: ID does not exist"
Oct 30 12:39:18.343638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.343607    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"]
Oct 30 12:39:18.343763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.343651    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b15ec3-openshift-default-8c8c5d477-hcrhw"]
Oct 30 12:39:18.356541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356513    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356541    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356555    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356576    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356591    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356603    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98q22\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-kube-api-access-98q22\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356617    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.356757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.356629    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f3122-ec0f-401f-b8b8-e37640160a8e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:18.594539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:18.594503    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:19.858543 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:39:19.858505    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:39:20.272253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:20.272221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:20.345980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:20.345940    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" path="/var/lib/kubelet/pods/0c2f3122-ec0f-401f-b8b8-e37640160a8e/volumes"
Oct 30 12:39:21.500817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:21.500777    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:21.651213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:21.651175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:22.267640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:22.267595    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.219922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:23.219883    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.398836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:23.398796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.660166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:23.660084    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"]
Oct 30 12:39:23.694673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:23.694640    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"]
Oct 30 12:39:23.831527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:23.831490    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.969797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:23.969761    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:24.183181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:24.183144    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:24.655600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:24.655568    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:24.896704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:24.896656    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:26.065983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:26.065943    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:27.852002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:27.851969    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:28.660947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:28.660904    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerName="payload-processing" containerID="cri-o://e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec" gracePeriod=25
Oct 30 12:39:28.695606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:28.695569    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerName="payload-pre-processing" containerID="cri-o://5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f" gracePeriod=25
Oct 30 12:39:29.332061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:29.332014    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:29.936482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:29.936427    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:30.059526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:30.059497    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"]
Oct 30 12:39:30.072029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:30.071972    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-933195-d7d5s"]
Oct 30 12:39:30.095004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:30.094866    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:30.337651 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:30.337612    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" path="/var/lib/kubelet/pods/d87f63ba-62ae-43cc-8abd-7afe63ca3f8c/volumes"
Oct 30 12:39:30.605661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:30.605572    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:30.680430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:30.680388    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:39:31.688782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:31.688748    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:39:32.054531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:32.054496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:33.843467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:33.843415    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:35.628740 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:35.628700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:37.163240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:37.163142    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:37.312573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:37.312523    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:37.454960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:37.454929    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"]
Oct 30 12:39:37.455196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:37.455173    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy" containerID="cri-o://0fc95420432a40ceb3e9bf5c4fcee1567774b6264952ae8eb55bb40252870895" gracePeriod=30
Oct 30 12:39:37.464378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:37.464343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:37.621488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:37.611393    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:38.051480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.051433    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"]
Oct 30 12:39:38.052113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052099    5060 cpu_manager.go:414] "containerMap: removing container" podUID="216b72af-eff4-484e-8e8e-236829576c71" containerName="revoke-keys"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052114    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216b72af-eff4-484e-8e8e-236829576c71" containerName="revoke-keys"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052127    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" containerName="revoke-keys"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052134    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" containerName="revoke-keys"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052153    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052159    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052165    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy"
Oct 30 12:39:38.052187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052170    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy"
Oct 30 12:39:38.052421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052265    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="216b72af-eff4-484e-8e8e-236829576c71" containerName="revoke-keys"
Oct 30 12:39:38.052421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052274    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87f63ba-62ae-43cc-8abd-7afe63ca3f8c" containerName="revoke-keys"
Oct 30 12:39:38.052421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052282    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy"
Oct 30 12:39:38.052421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.052289    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2f3122-ec0f-401f-b8b8-e37640160a8e" containerName="istio-proxy"
Oct 30 12:39:38.056077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.056059    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.072627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.072598    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"]
Oct 30 12:39:38.080790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.080512    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:39:38.080790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.080443    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:39:38.125302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.125270    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:39:38.125497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.125349    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:39:38.249637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.249606    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.250110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.249895    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.250110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.249950    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnnr\" (UniqueName: \"kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.351482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.351375    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.351482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.351416    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnnr\" (UniqueName: \"kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.351686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.351507    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.352034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.352008    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.354019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.353998    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.374922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.374893    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnnr\" (UniqueName: \"kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.670164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.670064    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:38.979839 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:39:38.979806    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a0ccc6_cf84_4819_8079_c3bc6948cda5.slice/crio-17a18c0cdfdbe8cd086f211f2d2515aad70e808847e32136f18ae2b63516880b WatchSource:0}: Error finding container 17a18c0cdfdbe8cd086f211f2d2515aad70e808847e32136f18ae2b63516880b: Status 404 returned error can't find the container with id 17a18c0cdfdbe8cd086f211f2d2515aad70e808847e32136f18ae2b63516880b
Oct 30 12:39:38.984567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:38.984529    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"]
Oct 30 12:39:39.314484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:39.314340    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:39.564329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:39.564283    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx" event={"ID":"91a0ccc6-cf84-4819-8079-c3bc6948cda5","Type":"ContainerStarted","Data":"1e4de927b23b735048970da3a6254556d125ef478739f245ad261bfd50c44dd9"}
Oct 30 12:39:39.564613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:39.564346    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx" event={"ID":"91a0ccc6-cf84-4819-8079-c3bc6948cda5","Type":"ContainerStarted","Data":"17a18c0cdfdbe8cd086f211f2d2515aad70e808847e32136f18ae2b63516880b"}
Oct 30 12:39:39.608782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:39.608721    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx" podStartSLOduration=1.608705186 podStartE2EDuration="1.608705186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:39.605088966 +0000 UTC m=+4183.399973460" watchObservedRunningTime="2026-10-30 12:39:39.608705186 +0000 UTC m=+4183.403589679"
Oct 30 12:39:40.574305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:40.574270    5060 generic.go:356] "Generic (PLEG): container finished" podID="91a0ccc6-cf84-4819-8079-c3bc6948cda5" containerID="1e4de927b23b735048970da3a6254556d125ef478739f245ad261bfd50c44dd9" exitCode=0
Oct 30 12:39:40.574704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:40.574403    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx" event={"ID":"91a0ccc6-cf84-4819-8079-c3bc6948cda5","Type":"ContainerDied","Data":"1e4de927b23b735048970da3a6254556d125ef478739f245ad261bfd50c44dd9"}
Oct 30 12:39:40.680274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:40.680236    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:39:41.213732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.213695    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:41.691750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.689393    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:39:41.760301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.760274    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:41.816499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.810138    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca\") pod \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") "
Oct 30 12:39:41.816499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.810683    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr\" (UniqueName: \"kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr\") pod \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") "
Oct 30 12:39:41.816499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.810741    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca\") pod \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\" (UID: \"91a0ccc6-cf84-4819-8079-c3bc6948cda5\") "
Oct 30 12:39:41.816499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.812033    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca" pod "91a0ccc6-cf84-4819-8079-c3bc6948cda5" (UID: "91a0ccc6-cf84-4819-8079-c3bc6948cda5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:41.818872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.817530    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr" pod "91a0ccc6-cf84-4819-8079-c3bc6948cda5" (UID: "91a0ccc6-cf84-4819-8079-c3bc6948cda5"). InnerVolumeSpecName "kube-api-access-bpnnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:41.818872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.817585    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca" pod "91a0ccc6-cf84-4819-8079-c3bc6948cda5" (UID: "91a0ccc6-cf84-4819-8079-c3bc6948cda5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:39:41.913037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.912950    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpnnr\" (UniqueName: \"kubernetes.io/projected/91a0ccc6-cf84-4819-8079-c3bc6948cda5-kube-api-access-bpnnr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:41.913037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.912985    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:41.913037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:41.913003    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a0ccc6-cf84-4819-8079-c3bc6948cda5-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.595487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.595393    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"
Oct 30 12:39:42.595487 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.595423    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx" event={"ID":"91a0ccc6-cf84-4819-8079-c3bc6948cda5","Type":"ContainerDied","Data":"17a18c0cdfdbe8cd086f211f2d2515aad70e808847e32136f18ae2b63516880b"}
Oct 30 12:39:42.595729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.595490    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17a18c0cdfdbe8cd086f211f2d2515aad70e808847e32136f18ae2b63516880b"
Oct 30 12:39:42.598416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.598383    5060 generic.go:356] "Generic (PLEG): container finished" podID="7a395608-0ff6-42de-adb2-c425bae17902" containerID="0fc95420432a40ceb3e9bf5c4fcee1567774b6264952ae8eb55bb40252870895" exitCode=0
Oct 30 12:39:42.598588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.598467    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" event={"ID":"7a395608-0ff6-42de-adb2-c425bae17902","Type":"ContainerDied","Data":"0fc95420432a40ceb3e9bf5c4fcee1567774b6264952ae8eb55bb40252870895"}
Oct 30 12:39:42.598588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.598529    5060 scope.go:122] "RemoveContainer" containerID="d7ed3f075a69da122994c708abda8d8759ecf6cbea893dffb5abff5bd62ef319"
Oct 30 12:39:42.722639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.722606    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:39:42.824158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.824113    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.824364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.824218    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.824875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.824247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.825207 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.824919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.825309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.824444    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:42.825309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.824834    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:42.825443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.825314    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.825746 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.825725    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.826236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.826209    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.826583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.826557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:42.826703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.826686    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.826929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.826908    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:42.827086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.827068    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert\") pod \"7a395608-0ff6-42de-adb2-c425bae17902\" (UID: \"7a395608-0ff6-42de-adb2-c425bae17902\") "
Oct 30 12:39:42.827443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.827419    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:42.827813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.827785    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "kube-api-access-6bx87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:42.828120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828100    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.828213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828124    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bx87\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-kube-api-access-6bx87\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.828213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828140    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.828213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828153    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.828213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828167    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a395608-0ff6-42de-adb2-c425bae17902-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.828213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828182    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.828421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828376    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:39:42.828813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.828785    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:39:42.829813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.829790    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token" pod "7a395608-0ff6-42de-adb2-c425bae17902" (UID: "7a395608-0ff6-42de-adb2-c425bae17902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:42.929519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.929429    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a395608-0ff6-42de-adb2-c425bae17902-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.929519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.929491    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a395608-0ff6-42de-adb2-c425bae17902-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:42.929519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:42.929501    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a395608-0ff6-42de-adb2-c425bae17902-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:43.092427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:43.092385    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:43.608920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:43.608891    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"
Oct 30 12:39:43.608920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:43.608906    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn" event={"ID":"7a395608-0ff6-42de-adb2-c425bae17902","Type":"ContainerDied","Data":"d26bc7488912ed458d9572434cc29e01eeafbc395bed3e6d76aae865e4b8c050"}
Oct 30 12:39:43.609181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:43.608980    5060 scope.go:122] "RemoveContainer" containerID="0fc95420432a40ceb3e9bf5c4fcee1567774b6264952ae8eb55bb40252870895"
Oct 30 12:39:43.677733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:43.677698    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"]
Oct 30 12:39:43.690018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:43.689991    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-933195-openshift-default-77ff6d57-cnldn"]
Oct 30 12:39:44.337462 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:44.337418    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a395608-0ff6-42de-adb2-c425bae17902" path="/var/lib/kubelet/pods/7a395608-0ff6-42de-adb2-c425bae17902/volumes"
Oct 30 12:39:44.774535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:44.774492    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:46.276071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:46.275960    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:46.787358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:46.787321    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:46.939857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:46.939808    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"]
Oct 30 12:39:46.948596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:46.948566    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-26b7d22d-8b9n5"]
Oct 30 12:39:47.001742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:47.001710    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:47.267696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:47.267658    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:47.462982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:47.462944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:47.623303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:47.623195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:48.032965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:48.032923    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"]
Oct 30 12:39:48.045698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:48.045665    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21b2303-ht656"]
Oct 30 12:39:48.338952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:48.338867    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216b72af-eff4-484e-8e8e-236829576c71" path="/var/lib/kubelet/pods/216b72af-eff4-484e-8e8e-236829576c71/volumes"
Oct 30 12:39:48.339647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:48.339629    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997c97d3-902a-4f6c-876c-a69814e8beeb" path="/var/lib/kubelet/pods/997c97d3-902a-4f6c-876c-a69814e8beeb/volumes"
Oct 30 12:39:48.722690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:48.722647    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:48.924194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:48.924090    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:49.031050 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.031024    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:39:49.087357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.087330    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:39:49.103918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.103883    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:49.126497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.126464    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp\" (UniqueName: \"kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp\") pod \"a8e244ec-eeec-4b46-bd38-392aeac08f65\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") "
Oct 30 12:39:49.126748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.126726    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume\") pod \"a8e244ec-eeec-4b46-bd38-392aeac08f65\" (UID: \"a8e244ec-eeec-4b46-bd38-392aeac08f65\") "
Oct 30 12:39:49.127187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.127101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume" pod "a8e244ec-eeec-4b46-bd38-392aeac08f65" (UID: "a8e244ec-eeec-4b46-bd38-392aeac08f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:49.127388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.127287    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e244ec-eeec-4b46-bd38-392aeac08f65-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:49.128802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.128775    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp" pod "a8e244ec-eeec-4b46-bd38-392aeac08f65" (UID: "a8e244ec-eeec-4b46-bd38-392aeac08f65"). InnerVolumeSpecName "kube-api-access-ctdnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:49.228229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.228198    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume\") pod \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") "
Oct 30 12:39:49.228480 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.228375    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29\" (UniqueName: \"kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29\") pod \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\" (UID: \"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd\") "
Oct 30 12:39:49.228697 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.228670    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume" pod "87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" (UID: "87de9ccf-f4be-4360-a5f4-bafa41fd8dbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:49.228878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.228861    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdnp\" (UniqueName: \"kubernetes.io/projected/a8e244ec-eeec-4b46-bd38-392aeac08f65-kube-api-access-ctdnp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:49.228959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.228881    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:49.231005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.230978    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29" pod "87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" (UID: "87de9ccf-f4be-4360-a5f4-bafa41fd8dbd"). InnerVolumeSpecName "kube-api-access-nwl29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:49.337478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.333599    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwl29\" (UniqueName: \"kubernetes.io/projected/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd-kube-api-access-nwl29\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:39:49.670045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.669957    5060 generic.go:356] "Generic (PLEG): container finished" podID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerID="e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec" exitCode=0
Oct 30 12:39:49.670045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.670002    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"
Oct 30 12:39:49.670303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.670041    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" event={"ID":"a8e244ec-eeec-4b46-bd38-392aeac08f65","Type":"ContainerDied","Data":"e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec"}
Oct 30 12:39:49.670303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.670109    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv" event={"ID":"a8e244ec-eeec-4b46-bd38-392aeac08f65","Type":"ContainerDied","Data":"02f68a57ee69060ce995c465c0925f7bbf60b7b2b3de000b016cebf3a98fd3f8"}
Oct 30 12:39:49.670303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.670143    5060 scope.go:122] "RemoveContainer" containerID="e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec"
Oct 30 12:39:49.671689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.671656    5060 generic.go:356] "Generic (PLEG): container finished" podID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerID="5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f" exitCode=0
Oct 30 12:39:49.671828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.671709    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"
Oct 30 12:39:49.671828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.671745    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" event={"ID":"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd","Type":"ContainerDied","Data":"5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f"}
Oct 30 12:39:49.671828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.671807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn" event={"ID":"87de9ccf-f4be-4360-a5f4-bafa41fd8dbd","Type":"ContainerDied","Data":"2301182ab9c44b3110defb1e1774f6ecfb8314c149b4ec719d0079b47cb5e25b"}
Oct 30 12:39:49.693212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.693194    5060 scope.go:122] "RemoveContainer" containerID="e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec"
Oct 30 12:39:49.693539 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:39:49.693517    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec\": container with ID starting with e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec not found: ID does not exist" containerID="e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec"
Oct 30 12:39:49.693622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.693547    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec"} err="failed to get container status \"e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec\": rpc error: code = NotFound desc = could not find container \"e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec\": container with ID starting with e2bce2f355dc8ab1c9c4ac07803839367774423a62c5f4cee8d98d8db561d2ec not found: ID does not exist"
Oct 30 12:39:49.693622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.693563    5060 scope.go:122] "RemoveContainer" containerID="5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f"
Oct 30 12:39:49.703493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.703467    5060 scope.go:122] "RemoveContainer" containerID="5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f"
Oct 30 12:39:49.703801 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:39:49.703778    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f\": container with ID starting with 5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f not found: ID does not exist" containerID="5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f"
Oct 30 12:39:49.703877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.703814    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f"} err="failed to get container status \"5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f\": rpc error: code = NotFound desc = could not find container \"5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f\": container with ID starting with 5c2f42ed683d5aa35b467216fb3491f662c42685c5b9a0584d61ccf6375cbb8f not found: ID does not exist"
Oct 30 12:39:49.736093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.736060    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"]
Oct 30 12:39:49.748716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.748684    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-933195-5ddb44fb8c-b5cxn"]
Oct 30 12:39:49.785857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.785827    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"]
Oct 30 12:39:49.789948 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:49.789923    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-933195-5c9ff88b66-qc6lv"]
Oct 30 12:39:49.858036 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:39:49.858002    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:39:50.335943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:50.335913    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" path="/var/lib/kubelet/pods/87de9ccf-f4be-4360-a5f4-bafa41fd8dbd/volumes"
Oct 30 12:39:50.336357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:50.336343    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" path="/var/lib/kubelet/pods/a8e244ec-eeec-4b46-bd38-392aeac08f65/volumes"
Oct 30 12:39:50.864256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:50.864223    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:52.693999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:52.693959    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:54.655557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:54.655525    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:54.899331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:54.899279    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:55.041081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:55.041043    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:55.289183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:55.289146    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:56.428316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:56.428278    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:56.584005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:56.583968    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:57.692906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:57.692862    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:58.038872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.038822    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:58.138475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.136347    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:39:58.146542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.144732    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:39:58.523476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.522653    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:58.670077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.670039    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"]
Oct 30 12:39:58.671426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.671389    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:58.682401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:58.682356    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bbae89-9r8wx"]
Oct 30 12:39:59.060400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:59.060364    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"]
Oct 30 12:39:59.072490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:59.072446    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d541f9-cb8cw"]
Oct 30 12:39:59.234603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:39:59.234547    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:00.336258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:00.336226    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a0ccc6-cf84-4819-8079-c3bc6948cda5" path="/var/lib/kubelet/pods/91a0ccc6-cf84-4819-8079-c3bc6948cda5/volumes"
Oct 30 12:40:00.336722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:00.336707    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c02e1df-4394-4588-b6c0-c417fd4f2bd9" path="/var/lib/kubelet/pods/9c02e1df-4394-4588-b6c0-c417fd4f2bd9/volumes"
Oct 30 12:40:00.689209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:00.689102    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:00.858305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:00.858269    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:02.557216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:02.557171    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:03.424767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:03.424726    5060 scope.go:122] "RemoveContainer" containerID="7971f57dd8ad86f44c30cc50f57914c553156dc72d39f7f29a632fa0a77387b5"
Oct 30 12:40:03.436292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:03.436265    5060 scope.go:122] "RemoveContainer" containerID="4d5b9a3a2dc17c09c53d2077f1e933dd1ca154e979a188702a8e24443e9258a1"
Oct 30 12:40:03.446593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:03.446567    5060 scope.go:122] "RemoveContainer" containerID="77244d96e46de8d0fbb4a7c33b90481f121b5ab60d59f8c3e909a15408e9808d"
Oct 30 12:40:04.346282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:04.346242    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:05.812835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:05.812793    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:05.974475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:05.974007    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.164360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.164185    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"]
Oct 30 12:40:06.165070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.165033    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy" containerID="cri-o://9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442" gracePeriod=30
Oct 30 12:40:06.222739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.222704    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.377509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.374897    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.821541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.821502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.988586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.988539    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"]
Oct 30 12:40:06.989212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989191    5060 cpu_manager.go:414] "containerMap: removing container" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerName="payload-pre-processing"
Oct 30 12:40:06.989212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989211    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerName="payload-pre-processing"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989236    5060 cpu_manager.go:414] "containerMap: removing container" podUID="91a0ccc6-cf84-4819-8079-c3bc6948cda5" containerName="revoke-keys"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989243    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a0ccc6-cf84-4819-8079-c3bc6948cda5" containerName="revoke-keys"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989257    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerName="payload-processing"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989261    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerName="payload-processing"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989269    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989274    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989286    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989290    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989380    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e244ec-eeec-4b46-bd38-392aeac08f65" containerName="payload-processing"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989388    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989401    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="87de9ccf-f4be-4360-a5f4-bafa41fd8dbd" containerName="payload-pre-processing"
Oct 30 12:40:06.989403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.989409    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a0ccc6-cf84-4819-8079-c3bc6948cda5" containerName="revoke-keys"
Oct 30 12:40:06.993147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:06.993127    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.012362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.012336    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"]
Oct 30 12:40:07.020395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.020369    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:40:07.020587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.020482    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:40:07.021729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.021711    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:40:07.026620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.026602    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:40:07.131878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.131789    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.132065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.131952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8dj\" (UniqueName: \"kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.132065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.131993    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.233713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.233670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8dj\" (UniqueName: \"kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.233938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.233746    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.233938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.233916    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.234745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.234717    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.236329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.236306    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.257885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.257855    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8dj\" (UniqueName: \"kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj\") pod \"maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.307654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.307616    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:07.629201 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:07.629166    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091da81a_7800_496f_9b0a_249595b44f32.slice/crio-18311542b5322c2a300b57e0fdbf5fb03a420f9762f4d5167d2b6d216e6fa000 WatchSource:0}: Error finding container 18311542b5322c2a300b57e0fdbf5fb03a420f9762f4d5167d2b6d216e6fa000: Status 404 returned error can't find the container with id 18311542b5322c2a300b57e0fdbf5fb03a420f9762f4d5167d2b6d216e6fa000
Oct 30 12:40:07.632037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.632006    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"]
Oct 30 12:40:07.849081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:07.849050    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw" event={"ID":"091da81a-7800-496f-9b0a-249595b44f32","Type":"ContainerStarted","Data":"18311542b5322c2a300b57e0fdbf5fb03a420f9762f4d5167d2b6d216e6fa000"}
Oct 30 12:40:08.537295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:08.537260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:08.860959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:08.860873    5060 generic.go:356] "Generic (PLEG): container finished" podID="091da81a-7800-496f-9b0a-249595b44f32" containerID="b997af5b8b0ce762c50d7fe5220ee9901ffe4b792379aa91865cd0a517e35301" exitCode=0
Oct 30 12:40:08.861392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:08.860959    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw" event={"ID":"091da81a-7800-496f-9b0a-249595b44f32","Type":"ContainerDied","Data":"b997af5b8b0ce762c50d7fe5220ee9901ffe4b792379aa91865cd0a517e35301"}
Oct 30 12:40:09.362073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.362039    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"]
Oct 30 12:40:09.362935 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.362917    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a395608-0ff6-42de-adb2-c425bae17902" containerName="istio-proxy"
Oct 30 12:40:09.366484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.366466    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.385282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.385260    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"]
Oct 30 12:40:09.465588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.465552    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.465788 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.465771    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.466084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.466058    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqgx\" (UniqueName: \"kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.569504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.569463    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqgx\" (UniqueName: \"kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.569737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.569547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.569737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.569661    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.574081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.574047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.574913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.574888    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.595986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.595959    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqgx\" (UniqueName: \"kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx\") pod \"maas-api-revoke-sub-4349d059-42krn\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:09.684934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:09.684850    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:10.004779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.004745    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"]
Oct 30 12:40:10.010153 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:10.010129    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fa000d_bf58_4d5f_9e83_4f0b5a056a0c.slice/crio-49ae37a93be37e17c79edc7277cdae5ed49ee66a2f106eb6e66ddde6126ae33a WatchSource:0}: Error finding container 49ae37a93be37e17c79edc7277cdae5ed49ee66a2f106eb6e66ddde6126ae33a: Status 404 returned error can't find the container with id 49ae37a93be37e17c79edc7277cdae5ed49ee66a2f106eb6e66ddde6126ae33a
Oct 30 12:40:10.035328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.035299    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:10.179277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.179247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca\") pod \"091da81a-7800-496f-9b0a-249595b44f32\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") "
Oct 30 12:40:10.179474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.179438    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca\") pod \"091da81a-7800-496f-9b0a-249595b44f32\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") "
Oct 30 12:40:10.179561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.179533    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj\" (UniqueName: \"kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj\") pod \"091da81a-7800-496f-9b0a-249595b44f32\" (UID: \"091da81a-7800-496f-9b0a-249595b44f32\") "
Oct 30 12:40:10.179625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.179596    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca" pod "091da81a-7800-496f-9b0a-249595b44f32" (UID: "091da81a-7800-496f-9b0a-249595b44f32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:10.180101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.180077    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091da81a-7800-496f-9b0a-249595b44f32-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:10.182048 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.182022    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca" pod "091da81a-7800-496f-9b0a-249595b44f32" (UID: "091da81a-7800-496f-9b0a-249595b44f32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:10.182238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.182215    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj" pod "091da81a-7800-496f-9b0a-249595b44f32" (UID: "091da81a-7800-496f-9b0a-249595b44f32"). InnerVolumeSpecName "kube-api-access-5s8dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:10.281513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.281482    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091da81a-7800-496f-9b0a-249595b44f32-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:10.281513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.281508    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8dj\" (UniqueName: \"kubernetes.io/projected/091da81a-7800-496f-9b0a-249595b44f32-kube-api-access-5s8dj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:10.473995 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.473892    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:10.879589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.879490    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw" event={"ID":"091da81a-7800-496f-9b0a-249595b44f32","Type":"ContainerDied","Data":"18311542b5322c2a300b57e0fdbf5fb03a420f9762f4d5167d2b6d216e6fa000"}
Oct 30 12:40:10.879589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.879528    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"
Oct 30 12:40:10.879811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.879588    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18311542b5322c2a300b57e0fdbf5fb03a420f9762f4d5167d2b6d216e6fa000"
Oct 30 12:40:10.881486 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.881420    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn" event={"ID":"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c","Type":"ContainerStarted","Data":"4ff70b8b6da82262701b439c2afc2480662ac68770a7c0a9459264d7cb3bf4d5"}
Oct 30 12:40:10.881615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.881487    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn" event={"ID":"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c","Type":"ContainerStarted","Data":"49ae37a93be37e17c79edc7277cdae5ed49ee66a2f106eb6e66ddde6126ae33a"}
Oct 30 12:40:10.932275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:10.932230    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn" podStartSLOduration=1.932215819 podStartE2EDuration="1.932215819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:10.928209443 +0000 UTC m=+4214.723093942" watchObservedRunningTime="2026-10-30 12:40:10.932215819 +0000 UTC m=+4214.727100313"
Oct 30 12:40:11.451668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.451639    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:40:11.613194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613229    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613285    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613318    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613348    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613519    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:11.613562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613540    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613547    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:11.613776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613604    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613634    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy\") pod \"96682ef0-5f44-4ee2-96be-59d56a7099a4\" (UID: \"96682ef0-5f44-4ee2-96be-59d56a7099a4\") "
Oct 30 12:40:11.613776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613645    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:11.613776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613747    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:11.614014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.613820    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:11.614422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.614281    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.614602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.614474    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.614602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.614499    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96682ef0-5f44-4ee2-96be-59d56a7099a4-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.614602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.614513    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.614602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.614528    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.616091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.616064    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "kube-api-access-wwz4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:11.616213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.616125    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:11.616317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.616290    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:40:11.616385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.616340    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token" pod "96682ef0-5f44-4ee2-96be-59d56a7099a4" (UID: "96682ef0-5f44-4ee2-96be-59d56a7099a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:11.641680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.641654    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"]
Oct 30 12:40:11.716210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.716137    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.716210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.716166    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwz4h\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-kube-api-access-wwz4h\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.716210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.716177    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.716210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.716186    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96682ef0-5f44-4ee2-96be-59d56a7099a4-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:11.772624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.772594    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:11.890739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.890699    5060 generic.go:356] "Generic (PLEG): container finished" podID="67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" containerID="4ff70b8b6da82262701b439c2afc2480662ac68770a7c0a9459264d7cb3bf4d5" exitCode=0
Oct 30 12:40:11.890910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.890777    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn" event={"ID":"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c","Type":"ContainerDied","Data":"4ff70b8b6da82262701b439c2afc2480662ac68770a7c0a9459264d7cb3bf4d5"}
Oct 30 12:40:11.892711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.892689    5060 generic.go:356] "Generic (PLEG): container finished" podID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerID="9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442" exitCode=0
Oct 30 12:40:11.892820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.892718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" event={"ID":"96682ef0-5f44-4ee2-96be-59d56a7099a4","Type":"ContainerDied","Data":"9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442"}
Oct 30 12:40:11.892820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.892745    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"
Oct 30 12:40:11.892820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.892769    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm" event={"ID":"96682ef0-5f44-4ee2-96be-59d56a7099a4","Type":"ContainerDied","Data":"7c30722ec7930095201d6bb2cb020c19854a5ccc462d4174f0237743f0312664"}
Oct 30 12:40:11.892820 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.892799    5060 scope.go:122] "RemoveContainer" containerID="9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442"
Oct 30 12:40:11.906355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.906336    5060 scope.go:122] "RemoveContainer" containerID="33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"
Oct 30 12:40:11.924891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.924867    5060 scope.go:122] "RemoveContainer" containerID="9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442"
Oct 30 12:40:11.925191 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:11.925163    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442\": container with ID starting with 9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442 not found: ID does not exist" containerID="9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442"
Oct 30 12:40:11.925258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.925202    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442"} err="failed to get container status \"9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442\": rpc error: code = NotFound desc = could not find container \"9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442\": container with ID starting with 9467a8f215d0e95ba255262e57a8293286d079bca2f141d40f657e76a6420442 not found: ID does not exist"
Oct 30 12:40:11.925258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.925220    5060 scope.go:122] "RemoveContainer" containerID="33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"
Oct 30 12:40:11.925526 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:11.925504    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d\": container with ID starting with 33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d not found: ID does not exist" containerID="33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"
Oct 30 12:40:11.925609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.925536    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d"} err="failed to get container status \"33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d\": rpc error: code = NotFound desc = could not find container \"33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d\": container with ID starting with 33f517e3085e3f2682a24c19ffa4576222df88e3e506d12b35dd73ad898e1a1d not found: ID does not exist"
Oct 30 12:40:11.975483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.975376    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"]
Oct 30 12:40:11.985183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:11.985148    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bbae89-openshift-default-79685bd9cc-vldfm"]
Oct 30 12:40:12.337395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:12.337322    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" path="/var/lib/kubelet/pods/96682ef0-5f44-4ee2-96be-59d56a7099a4/volumes"
Oct 30 12:40:12.464652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:12.464607    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:13.077807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.077779    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:13.238039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.238004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx\" (UniqueName: \"kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx\") pod \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") "
Oct 30 12:40:13.238250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.238067    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca\") pod \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") "
Oct 30 12:40:13.238330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.238262    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca\") pod \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\" (UID: \"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c\") "
Oct 30 12:40:13.238444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.238422    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca" pod "67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" (UID: "67fa000d-bf58-4d5f-9e83-4f0b5a056a0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:13.238707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.238688    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:13.240390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.240365    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca" pod "67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" (UID: "67fa000d-bf58-4d5f-9e83-4f0b5a056a0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:13.240542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.240504    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx" pod "67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" (UID: "67fa000d-bf58-4d5f-9e83-4f0b5a056a0c"). InnerVolumeSpecName "kube-api-access-grqgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:13.339462 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.339421    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:13.339638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.339486    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqgx\" (UniqueName: \"kubernetes.io/projected/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c-kube-api-access-grqgx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:13.918662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.918612    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn" event={"ID":"67fa000d-bf58-4d5f-9e83-4f0b5a056a0c","Type":"ContainerDied","Data":"49ae37a93be37e17c79edc7277cdae5ed49ee66a2f106eb6e66ddde6126ae33a"}
Oct 30 12:40:13.918662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.918656    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"
Oct 30 12:40:13.919248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:13.918672    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ae37a93be37e17c79edc7277cdae5ed49ee66a2f106eb6e66ddde6126ae33a"
Oct 30 12:40:14.321756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:14.321718    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:15.992500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:15.992436    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:17.636597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:17.636556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:17.776983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:17.776937    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:18.586371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.586329    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"]
Oct 30 12:40:18.587300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587275    5060 cpu_manager.go:414] "containerMap: removing container" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy"
Oct 30 12:40:18.587300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587301    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy"
Oct 30 12:40:18.587532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587328    5060 cpu_manager.go:414] "containerMap: removing container" podUID="091da81a-7800-496f-9b0a-249595b44f32" containerName="revoke-keys"
Oct 30 12:40:18.587532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587337    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091da81a-7800-496f-9b0a-249595b44f32" containerName="revoke-keys"
Oct 30 12:40:18.587532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587375    5060 cpu_manager.go:414] "containerMap: removing container" podUID="67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" containerName="revoke-keys"
Oct 30 12:40:18.587532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587382    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" containerName="revoke-keys"
Oct 30 12:40:18.587764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587546    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" containerName="revoke-keys"
Oct 30 12:40:18.587764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587558    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy"
Oct 30 12:40:18.587764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.587577    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="091da81a-7800-496f-9b0a-249595b44f32" containerName="revoke-keys"
Oct 30 12:40:18.591829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.591808    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.607212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.607176    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.607380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.607264    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dq7\" (UniqueName: \"kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.607507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.607389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.611348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.611315    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"]
Oct 30 12:40:18.614843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.614816    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:40:18.615010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.614816    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:40:18.620446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.620423    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:40:18.620617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.620419    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:40:18.710896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.710855    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dq7\" (UniqueName: \"kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.711395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.710977    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.711395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.711090    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.711768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.711747    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.713714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.713689    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.735522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.735492    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dq7\" (UniqueName: \"kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7\") pod \"maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:18.916761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:18.916665    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:19.224550 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:19.224509    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d031d2a_d6a3_4edd_b0f7_024c91551d8d.slice/crio-bda3f84f2a210dcaeff438195088b4ea2e56d8174d5fcd6811b02c673bceb187 WatchSource:0}: Error finding container bda3f84f2a210dcaeff438195088b4ea2e56d8174d5fcd6811b02c673bceb187: Status 404 returned error can't find the container with id bda3f84f2a210dcaeff438195088b4ea2e56d8174d5fcd6811b02c673bceb187
Oct 30 12:40:19.227677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.227654    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"]
Oct 30 12:40:19.660496 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.660429    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:19.742744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.742660    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"]
Oct 30 12:40:19.743312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.743298    5060 cpu_manager.go:414] "containerMap: removing container" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy"
Oct 30 12:40:19.743359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.743314    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy"
Oct 30 12:40:19.743462 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.743442    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="96682ef0-5f44-4ee2-96be-59d56a7099a4" containerName="istio-proxy"
Oct 30 12:40:19.747189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.747164    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.764954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.764926    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"]
Oct 30 12:40:19.827516 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.827479    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.827694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.827540    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpq5\" (UniqueName: \"kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.827804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.827790    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.859276 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:19.859238    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:40:19.928792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.928758    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.929018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.928823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.929018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.928851    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpq5\" (UniqueName: \"kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.929508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.929447    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.931407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.931386    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.958773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.958737    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpq5\" (UniqueName: \"kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5\") pod \"maas-api-revoke-sub-64ed7099-nbrph\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:19.969275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.969241    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c" event={"ID":"5d031d2a-d6a3-4edd-b0f7-024c91551d8d","Type":"ContainerStarted","Data":"279be4a20b5bca58c90d8b1e977db0a39d9e7f842fa19a641501702e2ccd9fa1"}
Oct 30 12:40:19.969400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:19.969298    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c" event={"ID":"5d031d2a-d6a3-4edd-b0f7-024c91551d8d","Type":"ContainerStarted","Data":"bda3f84f2a210dcaeff438195088b4ea2e56d8174d5fcd6811b02c673bceb187"}
Oct 30 12:40:20.021728 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.021606    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c" podStartSLOduration=2.02158967 podStartE2EDuration="2.02158967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:20.018678251 +0000 UTC m=+4223.813562745" watchObservedRunningTime="2026-10-30 12:40:20.02158967 +0000 UTC m=+4223.816474164"
Oct 30 12:40:20.061750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.061710    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:20.389619 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:20.389584    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d3b44f_0152_4ccf_90db_323541ed1c02.slice/crio-d80c69a6536e2f1c81b8b7cfd547a9a7767436b73e7c3ecdc054d3b27aca4e9f WatchSource:0}: Error finding container d80c69a6536e2f1c81b8b7cfd547a9a7767436b73e7c3ecdc054d3b27aca4e9f: Status 404 returned error can't find the container with id d80c69a6536e2f1c81b8b7cfd547a9a7767436b73e7c3ecdc054d3b27aca4e9f
Oct 30 12:40:20.391034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.391010    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"]
Oct 30 12:40:20.981405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.981309    5060 generic.go:356] "Generic (PLEG): container finished" podID="5d031d2a-d6a3-4edd-b0f7-024c91551d8d" containerID="279be4a20b5bca58c90d8b1e977db0a39d9e7f842fa19a641501702e2ccd9fa1" exitCode=0
Oct 30 12:40:20.981890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.981396    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c" event={"ID":"5d031d2a-d6a3-4edd-b0f7-024c91551d8d","Type":"ContainerDied","Data":"279be4a20b5bca58c90d8b1e977db0a39d9e7f842fa19a641501702e2ccd9fa1"}
Oct 30 12:40:20.983547 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.983523    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph" event={"ID":"f9d3b44f-0152-4ccf-90db-323541ed1c02","Type":"ContainerStarted","Data":"9df12acfc13c5e5cd7c4d597cf0e9201a0032198da7ec07fa4686a90c2fbe3df"}
Oct 30 12:40:20.983668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:20.983568    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph" event={"ID":"f9d3b44f-0152-4ccf-90db-323541ed1c02","Type":"ContainerStarted","Data":"d80c69a6536e2f1c81b8b7cfd547a9a7767436b73e7c3ecdc054d3b27aca4e9f"}
Oct 30 12:40:21.073614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:21.073557    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph" podStartSLOduration=2.0735405670000002 podStartE2EDuration="2.073540567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:21.069847494 +0000 UTC m=+4224.864731990" watchObservedRunningTime="2026-10-30 12:40:21.073540567 +0000 UTC m=+4224.868425101"
Oct 30 12:40:21.568518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:21.568479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:21.995352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:21.995313    5060 generic.go:356] "Generic (PLEG): container finished" podID="f9d3b44f-0152-4ccf-90db-323541ed1c02" containerID="9df12acfc13c5e5cd7c4d597cf0e9201a0032198da7ec07fa4686a90c2fbe3df" exitCode=0
Oct 30 12:40:21.995834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:21.995401    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph" event={"ID":"f9d3b44f-0152-4ccf-90db-323541ed1c02","Type":"ContainerDied","Data":"9df12acfc13c5e5cd7c4d597cf0e9201a0032198da7ec07fa4686a90c2fbe3df"}
Oct 30 12:40:22.178331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.178300    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:22.259045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.258943    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca\") pod \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") "
Oct 30 12:40:22.259234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.259082    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7\" (UniqueName: \"kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7\") pod \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") "
Oct 30 12:40:22.259234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.259149    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca\") pod \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\" (UID: \"5d031d2a-d6a3-4edd-b0f7-024c91551d8d\") "
Oct 30 12:40:22.259636 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.259603    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca" pod "5d031d2a-d6a3-4edd-b0f7-024c91551d8d" (UID: "5d031d2a-d6a3-4edd-b0f7-024c91551d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:22.261332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.261301    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca" pod "5d031d2a-d6a3-4edd-b0f7-024c91551d8d" (UID: "5d031d2a-d6a3-4edd-b0f7-024c91551d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:22.261739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.261711    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7" pod "5d031d2a-d6a3-4edd-b0f7-024c91551d8d" (UID: "5d031d2a-d6a3-4edd-b0f7-024c91551d8d"). InnerVolumeSpecName "kube-api-access-z4dq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:22.361586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.361539    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:22.361586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.361581    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:22.361789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:22.361596    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dq7\" (UniqueName: \"kubernetes.io/projected/5d031d2a-d6a3-4edd-b0f7-024c91551d8d-kube-api-access-z4dq7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:23.008386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.008348    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"
Oct 30 12:40:23.008992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.008961    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c" event={"ID":"5d031d2a-d6a3-4edd-b0f7-024c91551d8d","Type":"ContainerDied","Data":"bda3f84f2a210dcaeff438195088b4ea2e56d8174d5fcd6811b02c673bceb187"}
Oct 30 12:40:23.009772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.009747    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda3f84f2a210dcaeff438195088b4ea2e56d8174d5fcd6811b02c673bceb187"
Oct 30 12:40:23.173976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.173945    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:23.275836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.275734    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca\") pod \"f9d3b44f-0152-4ccf-90db-323541ed1c02\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") "
Oct 30 12:40:23.276026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.275834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca\") pod \"f9d3b44f-0152-4ccf-90db-323541ed1c02\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") "
Oct 30 12:40:23.276026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.275952    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5\" (UniqueName: \"kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5\") pod \"f9d3b44f-0152-4ccf-90db-323541ed1c02\" (UID: \"f9d3b44f-0152-4ccf-90db-323541ed1c02\") "
Oct 30 12:40:23.276181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.276101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca" pod "f9d3b44f-0152-4ccf-90db-323541ed1c02" (UID: "f9d3b44f-0152-4ccf-90db-323541ed1c02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:23.276602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.276561    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:23.280289 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.280265    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5" pod "f9d3b44f-0152-4ccf-90db-323541ed1c02" (UID: "f9d3b44f-0152-4ccf-90db-323541ed1c02"). InnerVolumeSpecName "kube-api-access-wwpq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:23.285585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.285557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca" pod "f9d3b44f-0152-4ccf-90db-323541ed1c02" (UID: "f9d3b44f-0152-4ccf-90db-323541ed1c02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:23.377872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.377830    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpq5\" (UniqueName: \"kubernetes.io/projected/f9d3b44f-0152-4ccf-90db-323541ed1c02-kube-api-access-wwpq5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:23.377872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.377873    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d3b44f-0152-4ccf-90db-323541ed1c02-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:23.470835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.470796    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:23.631743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:23.631639    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:24.034664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:24.034621    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph" event={"ID":"f9d3b44f-0152-4ccf-90db-323541ed1c02","Type":"ContainerDied","Data":"d80c69a6536e2f1c81b8b7cfd547a9a7767436b73e7c3ecdc054d3b27aca4e9f"}
Oct 30 12:40:24.035181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:24.034672    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"
Oct 30 12:40:24.035181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:24.034697    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80c69a6536e2f1c81b8b7cfd547a9a7767436b73e7c3ecdc054d3b27aca4e9f"
Oct 30 12:40:25.036890 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:25.036852    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"]
Oct 30 12:40:25.049037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:25.049005    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99c5b4-wrjxg"]
Oct 30 12:40:25.274940 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:25.274891    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:26.336412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.336381    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eccb7d8-55a3-4405-b5be-2684d495c893" path="/var/lib/kubelet/pods/5eccb7d8-55a3-4405-b5be-2684d495c893/volumes"
Oct 30 12:40:26.663039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.662951    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main" containerID="cri-o://a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e" gracePeriod=45
Oct 30 12:40:26.879015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.878984    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"]
Oct 30 12:40:26.879650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.879633    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f9d3b44f-0152-4ccf-90db-323541ed1c02" containerName="revoke-keys"
Oct 30 12:40:26.879650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.879650    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d3b44f-0152-4ccf-90db-323541ed1c02" containerName="revoke-keys"
Oct 30 12:40:26.879885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.879679    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5d031d2a-d6a3-4edd-b0f7-024c91551d8d" containerName="revoke-keys"
Oct 30 12:40:26.879885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.879685    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d031d2a-d6a3-4edd-b0f7-024c91551d8d" containerName="revoke-keys"
Oct 30 12:40:26.879885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.879827    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d3b44f-0152-4ccf-90db-323541ed1c02" containerName="revoke-keys"
Oct 30 12:40:26.879885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.879847    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d031d2a-d6a3-4edd-b0f7-024c91551d8d" containerName="revoke-keys"
Oct 30 12:40:26.883786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.883761    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:26.901415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.901373    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"]
Oct 30 12:40:26.907226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.907198    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"]
Oct 30 12:40:26.908549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.908497    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:40:26.911730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.911657    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:40:26.912066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.912028    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:40:26.912301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.912266    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:40:26.917361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.917300    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:26.939442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.939370    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"]
Oct 30 12:40:26.955112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.954971    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:26.955112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.955050    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6chl\" (UniqueName: \"kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:26.955267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.955170    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:26.979731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:26.979703    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 12:40:27.051388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.051351    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"]
Oct 30 12:40:27.056386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.056356    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir\") pod \"13d89c0c-fb4e-487e-899e-15edf1495e6e\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") "
Oct 30 12:40:27.057070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.057048    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs\") pod \"13d89c0c-fb4e-487e-899e-15edf1495e6e\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") "
Oct 30 12:40:27.057327 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.057311    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache\") pod \"13d89c0c-fb4e-487e-899e-15edf1495e6e\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") "
Oct 30 12:40:27.057521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.056977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir" pod "13d89c0c-fb4e-487e-899e-15edf1495e6e" (UID: "13d89c0c-fb4e-487e-899e-15edf1495e6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.057750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.057607    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7cde04-7xw2t"]
Oct 30 12:40:27.057934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.057763    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache" pod "13d89c0c-fb4e-487e-899e-15edf1495e6e" (UID: "13d89c0c-fb4e-487e-899e-15edf1495e6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.058113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.058092    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4\" (UniqueName: \"kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4\") pod \"13d89c0c-fb4e-487e-899e-15edf1495e6e\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") "
Oct 30 12:40:27.058301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.058288    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home\") pod \"13d89c0c-fb4e-487e-899e-15edf1495e6e\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") "
Oct 30 12:40:27.058696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.058670    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home" pod "13d89c0c-fb4e-487e-899e-15edf1495e6e" (UID: "13d89c0c-fb4e-487e-899e-15edf1495e6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.059176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.059161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm\") pod \"13d89c0c-fb4e-487e-899e-15edf1495e6e\" (UID: \"13d89c0c-fb4e-487e-899e-15edf1495e6e\") "
Oct 30 12:40:27.059562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.059543    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.060510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.060488    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6chl\" (UniqueName: \"kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.060814 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.060796    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.061415 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.061187    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs" pod "13d89c0c-fb4e-487e-899e-15edf1495e6e" (UID: "13d89c0c-fb4e-487e-899e-15edf1495e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:27.061617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.061598    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.065435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.064997    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4" pod "13d89c0c-fb4e-487e-899e-15edf1495e6e" (UID: "13d89c0c-fb4e-487e-899e-15edf1495e6e"). InnerVolumeSpecName "kube-api-access-vkdp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:27.065435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.065311    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.065827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.065699    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.065911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.065863    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brbx\" (UniqueName: \"kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.066088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.066040    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.066088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.066065    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.066088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.066079    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d89c0c-fb4e-487e-899e-15edf1495e6e-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.066240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.066091    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.066240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.066106    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkdp4\" (UniqueName: \"kubernetes.io/projected/13d89c0c-fb4e-487e-899e-15edf1495e6e-kube-api-access-vkdp4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.076942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.076886    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm" pod "13d89c0c-fb4e-487e-899e-15edf1495e6e" (UID: "13d89c0c-fb4e-487e-899e-15edf1495e6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.078518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.077856    5060 generic.go:356] "Generic (PLEG): container finished" podID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerID="a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e" exitCode=0
Oct 30 12:40:27.078518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.077998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" event={"ID":"13d89c0c-fb4e-487e-899e-15edf1495e6e","Type":"ContainerDied","Data":"a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e"}
Oct 30 12:40:27.078518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.078039    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x" event={"ID":"13d89c0c-fb4e-487e-899e-15edf1495e6e","Type":"ContainerDied","Data":"0b5666bd697dca4e3de9b5d6a75457e15794492983cca820065f86e664d01511"}
Oct 30 12:40:27.078518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.078059    5060 scope.go:122] "RemoveContainer" containerID="a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e"
Oct 30 12:40:27.078518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.078199    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"
Oct 30 12:40:27.080013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.079992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.097094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.097059    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6chl\" (UniqueName: \"kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl\") pod \"maas-api-revoke-sub-7db7b6d7-fdpg4\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.110246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.110219    5060 scope.go:122] "RemoveContainer" containerID="a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e"
Oct 30 12:40:27.110625 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:27.110604    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e\": container with ID starting with a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e not found: ID does not exist" containerID="a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e"
Oct 30 12:40:27.110718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.110635    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e"} err="failed to get container status \"a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e\": rpc error: code = NotFound desc = could not find container \"a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e\": container with ID starting with a8e05827fa3fac359c101e80e88045a522cc527c90c8eec8e9c5b3315b4a817e not found: ID does not exist"
Oct 30 12:40:27.144033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.143995    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"]
Oct 30 12:40:27.148003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.147970    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/simulator-w3-4e261b-kserve-686d99bd46-vnl9x"]
Oct 30 12:40:27.167304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.167273    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brbx\" (UniqueName: \"kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.167556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.167406    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.167556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.167540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.167708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.167631    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d89c0c-fb4e-487e-899e-15edf1495e6e-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.168237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.168173    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.170162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.170136    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.191007 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.190976    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brbx\" (UniqueName: \"kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx\") pod \"maas-api-revoke-sub-c8823713-lmr9q\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.203981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.203955    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:27.246420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.246390    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:27.423469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.423139    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:27.521153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.520970    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"]
Oct 30 12:40:27.524635 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:27.524584    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956f1d85_96a2_4dbb_a595_866c71f49724.slice/crio-f076a2840e3fa70a99f083ccb08fa54fbe2bc36e95ebfe3adf8664dee6e0efd0 WatchSource:0}: Error finding container f076a2840e3fa70a99f083ccb08fa54fbe2bc36e95ebfe3adf8664dee6e0efd0: Status 404 returned error can't find the container with id f076a2840e3fa70a99f083ccb08fa54fbe2bc36e95ebfe3adf8664dee6e0efd0
Oct 30 12:40:27.594991 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:27.594959    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5e6b5a_3fe3_47bd_948f_171587193952.slice/crio-61ec0f81f69b89a4a80a95faf324c5a627387d1becd5a30f22f57dc05342dfa1 WatchSource:0}: Error finding container 61ec0f81f69b89a4a80a95faf324c5a627387d1becd5a30f22f57dc05342dfa1: Status 404 returned error can't find the container with id 61ec0f81f69b89a4a80a95faf324c5a627387d1becd5a30f22f57dc05342dfa1
Oct 30 12:40:27.595685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:27.595556    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"]
Oct 30 12:40:28.107619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.107518    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4" event={"ID":"956f1d85-96a2-4dbb-a595-866c71f49724","Type":"ContainerStarted","Data":"be352c81d725beaf20e15fac11c96fe6d26aa9c1a05c8211e66d74195c5f40b9"}
Oct 30 12:40:28.107794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.107625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4" event={"ID":"956f1d85-96a2-4dbb-a595-866c71f49724","Type":"ContainerStarted","Data":"f076a2840e3fa70a99f083ccb08fa54fbe2bc36e95ebfe3adf8664dee6e0efd0"}
Oct 30 12:40:28.109512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.109441    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q" event={"ID":"6b5e6b5a-3fe3-47bd-948f-171587193952","Type":"ContainerStarted","Data":"7f294e93722e3f889a1a4bd3bb2eb18a3ec4895d9fa2962ac77ce8dc2eb67496"}
Oct 30 12:40:28.109643 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.109522    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q" event={"ID":"6b5e6b5a-3fe3-47bd-948f-171587193952","Type":"ContainerStarted","Data":"61ec0f81f69b89a4a80a95faf324c5a627387d1becd5a30f22f57dc05342dfa1"}
Oct 30 12:40:28.149324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.149267    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4" podStartSLOduration=2.1492521079999998 podStartE2EDuration="2.149252108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:28.14820925 +0000 UTC m=+4231.943093746" watchObservedRunningTime="2026-10-30 12:40:28.149252108 +0000 UTC m=+4231.944136601"
Oct 30 12:40:28.191882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.191824    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q" podStartSLOduration=2.191810405 podStartE2EDuration="2.191810405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:28.188248165 +0000 UTC m=+4231.983132660" watchObservedRunningTime="2026-10-30 12:40:28.191810405 +0000 UTC m=+4231.986694917"
Oct 30 12:40:28.336184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.336149    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" path="/var/lib/kubelet/pods/13d89c0c-fb4e-487e-899e-15edf1495e6e/volumes"
Oct 30 12:40:28.336866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:28.336849    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a176d82f-2ad6-4bbe-9937-71214a1d9521" path="/var/lib/kubelet/pods/a176d82f-2ad6-4bbe-9937-71214a1d9521/volumes"
Oct 30 12:40:29.024183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.024148    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:29.128675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.127989    5060 generic.go:356] "Generic (PLEG): container finished" podID="956f1d85-96a2-4dbb-a595-866c71f49724" containerID="be352c81d725beaf20e15fac11c96fe6d26aa9c1a05c8211e66d74195c5f40b9" exitCode=0
Oct 30 12:40:29.128675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.128605    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4" event={"ID":"956f1d85-96a2-4dbb-a595-866c71f49724","Type":"ContainerDied","Data":"be352c81d725beaf20e15fac11c96fe6d26aa9c1a05c8211e66d74195c5f40b9"}
Oct 30 12:40:29.130589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.130555    5060 generic.go:356] "Generic (PLEG): container finished" podID="6b5e6b5a-3fe3-47bd-948f-171587193952" containerID="7f294e93722e3f889a1a4bd3bb2eb18a3ec4895d9fa2962ac77ce8dc2eb67496" exitCode=0
Oct 30 12:40:29.130733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.130611    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q" event={"ID":"6b5e6b5a-3fe3-47bd-948f-171587193952","Type":"ContainerDied","Data":"7f294e93722e3f889a1a4bd3bb2eb18a3ec4895d9fa2962ac77ce8dc2eb67496"}
Oct 30 12:40:29.217554 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.216009    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:29.407543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.405387    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:29.987900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:29.987858    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:30.359604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.359575    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:30.404465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.404421    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:30.441716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.441680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:30.530406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530304    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca\") pod \"6b5e6b5a-3fe3-47bd-948f-171587193952\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") "
Oct 30 12:40:30.530406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530371    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca\") pod \"956f1d85-96a2-4dbb-a595-866c71f49724\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") "
Oct 30 12:40:30.530406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530402    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca\") pod \"6b5e6b5a-3fe3-47bd-948f-171587193952\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") "
Oct 30 12:40:30.530742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530578    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx\" (UniqueName: \"kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx\") pod \"6b5e6b5a-3fe3-47bd-948f-171587193952\" (UID: \"6b5e6b5a-3fe3-47bd-948f-171587193952\") "
Oct 30 12:40:30.530742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530618    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl\" (UniqueName: \"kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl\") pod \"956f1d85-96a2-4dbb-a595-866c71f49724\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") "
Oct 30 12:40:30.530852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca" pod "6b5e6b5a-3fe3-47bd-948f-171587193952" (UID: "6b5e6b5a-3fe3-47bd-948f-171587193952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:30.530852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530751    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca" pod "956f1d85-96a2-4dbb-a595-866c71f49724" (UID: "956f1d85-96a2-4dbb-a595-866c71f49724"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:30.530852 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.530818    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca\") pod \"956f1d85-96a2-4dbb-a595-866c71f49724\" (UID: \"956f1d85-96a2-4dbb-a595-866c71f49724\") "
Oct 30 12:40:30.531419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.531395    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.531564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.531422    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.533362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.533335    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl" pod "956f1d85-96a2-4dbb-a595-866c71f49724" (UID: "956f1d85-96a2-4dbb-a595-866c71f49724"). InnerVolumeSpecName "kube-api-access-v6chl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:30.533558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.533540    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca" pod "6b5e6b5a-3fe3-47bd-948f-171587193952" (UID: "6b5e6b5a-3fe3-47bd-948f-171587193952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:30.533638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.533553    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca" pod "956f1d85-96a2-4dbb-a595-866c71f49724" (UID: "956f1d85-96a2-4dbb-a595-866c71f49724"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:30.533638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.533607    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx" pod "6b5e6b5a-3fe3-47bd-948f-171587193952" (UID: "6b5e6b5a-3fe3-47bd-948f-171587193952"). InnerVolumeSpecName "kube-api-access-2brbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:30.578543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.578515    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"]
Oct 30 12:40:30.582527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.582502    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-737f64-v6z5c"]
Oct 30 12:40:30.583866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.583839    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:30.633150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.633118    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956f1d85-96a2-4dbb-a595-866c71f49724-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.633150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.633147    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5e6b5a-3fe3-47bd-948f-171587193952-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.633356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.633159    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2brbx\" (UniqueName: \"kubernetes.io/projected/6b5e6b5a-3fe3-47bd-948f-171587193952-kube-api-access-2brbx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.633356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.633169    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6chl\" (UniqueName: \"kubernetes.io/projected/956f1d85-96a2-4dbb-a595-866c71f49724-kube-api-access-v6chl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.686301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686266    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"]
Oct 30 12:40:30.686929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686912    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6b5e6b5a-3fe3-47bd-948f-171587193952" containerName="revoke-keys"
Oct 30 12:40:30.686929 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686929    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5e6b5a-3fe3-47bd-948f-171587193952" containerName="revoke-keys"
Oct 30 12:40:30.687080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686958    5060 cpu_manager.go:414] "containerMap: removing container" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main"
Oct 30 12:40:30.687080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686964    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main"
Oct 30 12:40:30.687080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686985    5060 cpu_manager.go:414] "containerMap: removing container" podUID="956f1d85-96a2-4dbb-a595-866c71f49724" containerName="revoke-keys"
Oct 30 12:40:30.687080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.686990    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956f1d85-96a2-4dbb-a595-866c71f49724" containerName="revoke-keys"
Oct 30 12:40:30.687080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.687071    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5e6b5a-3fe3-47bd-948f-171587193952" containerName="revoke-keys"
Oct 30 12:40:30.687080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.687081    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d89c0c-fb4e-487e-899e-15edf1495e6e" containerName="main"
Oct 30 12:40:30.687416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.687093    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="956f1d85-96a2-4dbb-a595-866c71f49724" containerName="revoke-keys"
Oct 30 12:40:30.690967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.690942    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.708949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.708923    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"]
Oct 30 12:40:30.835433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.835288    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.868871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.835426    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.868871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.835572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wjm\" (UniqueName: \"kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.936592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.936545    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.936777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.936650    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.936777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.936727    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wjm\" (UniqueName: \"kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.937293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.937274    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.939192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.939168    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:30.960183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:30.960153    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wjm\" (UniqueName: \"kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm\") pod \"maas-api-revoke-sub-9064398b-crzbq\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:31.010175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.010149    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:31.113122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.113003    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:31.153283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.153252    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"
Oct 30 12:40:31.153530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.153296    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4" event={"ID":"956f1d85-96a2-4dbb-a595-866c71f49724","Type":"ContainerDied","Data":"f076a2840e3fa70a99f083ccb08fa54fbe2bc36e95ebfe3adf8664dee6e0efd0"}
Oct 30 12:40:31.153530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.153340    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f076a2840e3fa70a99f083ccb08fa54fbe2bc36e95ebfe3adf8664dee6e0efd0"
Oct 30 12:40:31.155016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.154990    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q" event={"ID":"6b5e6b5a-3fe3-47bd-948f-171587193952","Type":"ContainerDied","Data":"61ec0f81f69b89a4a80a95faf324c5a627387d1becd5a30f22f57dc05342dfa1"}
Oct 30 12:40:31.155138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.155030    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ec0f81f69b89a4a80a95faf324c5a627387d1becd5a30f22f57dc05342dfa1"
Oct 30 12:40:31.155138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.155070    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"
Oct 30 12:40:31.316067 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:31.316034    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbefc45_82a8_4bba_b93d_05b8e8e1504d.slice/crio-4b6bc21f3042b4337997f9c6c437d2463687394e1a97a56f1b00dfa12f5eb461 WatchSource:0}: Error finding container 4b6bc21f3042b4337997f9c6c437d2463687394e1a97a56f1b00dfa12f5eb461: Status 404 returned error can't find the container with id 4b6bc21f3042b4337997f9c6c437d2463687394e1a97a56f1b00dfa12f5eb461
Oct 30 12:40:31.317905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:31.317881    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"]
Oct 30 12:40:32.164502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:32.164470    5060 generic.go:356] "Generic (PLEG): container finished" podID="adbefc45-82a8-4bba-b93d-05b8e8e1504d" containerID="78ff77129b1632b91d3afaef6111663a28f4ed6d26ad9b3d6f8ab87339b88454" exitCode=0
Oct 30 12:40:32.164862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:32.164544    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq" event={"ID":"adbefc45-82a8-4bba-b93d-05b8e8e1504d","Type":"ContainerDied","Data":"78ff77129b1632b91d3afaef6111663a28f4ed6d26ad9b3d6f8ab87339b88454"}
Oct 30 12:40:32.164862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:32.164601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq" event={"ID":"adbefc45-82a8-4bba-b93d-05b8e8e1504d","Type":"ContainerStarted","Data":"4b6bc21f3042b4337997f9c6c437d2463687394e1a97a56f1b00dfa12f5eb461"}
Oct 30 12:40:32.243519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:32.242357    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:32.335802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:32.335759    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d031d2a-d6a3-4edd-b0f7-024c91551d8d" path="/var/lib/kubelet/pods/5d031d2a-d6a3-4edd-b0f7-024c91551d8d/volumes"
Oct 30 12:40:33.365140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.365103    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:33.468955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.468921    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca\") pod \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") "
Oct 30 12:40:33.469153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.469101    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca\") pod \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") "
Oct 30 12:40:33.469205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.469171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm\" (UniqueName: \"kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm\") pod \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\" (UID: \"adbefc45-82a8-4bba-b93d-05b8e8e1504d\") "
Oct 30 12:40:33.469339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.469313    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca" pod "adbefc45-82a8-4bba-b93d-05b8e8e1504d" (UID: "adbefc45-82a8-4bba-b93d-05b8e8e1504d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:33.469514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.469494    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:33.471490 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.471432    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm" pod "adbefc45-82a8-4bba-b93d-05b8e8e1504d" (UID: "adbefc45-82a8-4bba-b93d-05b8e8e1504d"). InnerVolumeSpecName "kube-api-access-b8wjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:33.471768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.471750    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca" pod "adbefc45-82a8-4bba-b93d-05b8e8e1504d" (UID: "adbefc45-82a8-4bba-b93d-05b8e8e1504d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:33.570707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.570673    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wjm\" (UniqueName: \"kubernetes.io/projected/adbefc45-82a8-4bba-b93d-05b8e8e1504d-kube-api-access-b8wjm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:33.570707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:33.570705    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbefc45-82a8-4bba-b93d-05b8e8e1504d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:34.088725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:34.088689    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:34.182181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:34.182136    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq" event={"ID":"adbefc45-82a8-4bba-b93d-05b8e8e1504d","Type":"ContainerDied","Data":"4b6bc21f3042b4337997f9c6c437d2463687394e1a97a56f1b00dfa12f5eb461"}
Oct 30 12:40:34.182181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:34.182167    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"
Oct 30 12:40:34.182406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:34.182194    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6bc21f3042b4337997f9c6c437d2463687394e1a97a56f1b00dfa12f5eb461"
Oct 30 12:40:35.913464 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:35.913412    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:38.105406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:38.105369    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:39.188646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:39.188603    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:39.382757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:39.382723    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:39.575086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:39.575053    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"]
Oct 30 12:40:39.575323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:39.575293    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy" containerID="cri-o://0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231" gracePeriod=30
Oct 30 12:40:39.933275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:39.933179    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:40.415024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.414989    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"]
Oct 30 12:40:40.415592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.415577    5060 cpu_manager.go:414] "containerMap: removing container" podUID="adbefc45-82a8-4bba-b93d-05b8e8e1504d" containerName="revoke-keys"
Oct 30 12:40:40.415681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.415592    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbefc45-82a8-4bba-b93d-05b8e8e1504d" containerName="revoke-keys"
Oct 30 12:40:40.415839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.415827    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbefc45-82a8-4bba-b93d-05b8e8e1504d" containerName="revoke-keys"
Oct 30 12:40:40.419585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.419567    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.437647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.437626    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:40:40.449667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.449645    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:40:40.449787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.449677    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:40:40.449787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.449718    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:40:40.469236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.469213    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"]
Oct 30 12:40:40.474412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.474385    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmbs\" (UniqueName: \"kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.474587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.474526    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.474664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.474650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.576354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.576321    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.576600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.576415    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.576600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.576511    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmbs\" (UniqueName: \"kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.577099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.577078    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.578978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.578957    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.602406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.602378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmbs\" (UniqueName: \"kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:40.655084 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.655046    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:40.734937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:40.734900    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:41.054799 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:41.054768    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e170462_cccd_4451_9970_58003dc1a40b.slice/crio-e4c117a76d360e703871bd51079fd68ef5a34594922e88c7d40b1e09bc793300 WatchSource:0}: Error finding container e4c117a76d360e703871bd51079fd68ef5a34594922e88c7d40b1e09bc793300: Status 404 returned error can't find the container with id e4c117a76d360e703871bd51079fd68ef5a34594922e88c7d40b1e09bc793300
Oct 30 12:40:41.055964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.055941    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"]
Oct 30 12:40:41.246117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.246083    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885" event={"ID":"0e170462-cccd-4451-9970-58003dc1a40b","Type":"ContainerStarted","Data":"e4c117a76d360e703871bd51079fd68ef5a34594922e88c7d40b1e09bc793300"}
Oct 30 12:40:41.412396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.412359    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"]
Oct 30 12:40:41.416659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.416638    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.443673 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.443647    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"]
Oct 30 12:40:41.491072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.491030    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.491253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.491169    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmkq\" (UniqueName: \"kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.491368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.491350    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.592994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.592958    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.593161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.593059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmkq\" (UniqueName: \"kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.593227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.593201    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.593684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.593661    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.595627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.595606    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.628339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.628309    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmkq\" (UniqueName: \"kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq\") pod \"maas-api-revoke-sub-eabc5519-5cms2\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:41.730607 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:41.730571    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:42.056125 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:42.056093    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2fc0a2_cbfe_49ed_8941_89f04f7525c9.slice/crio-f1dce2859c686059393e355dca7c6d4f16e7c5ac938965ec13db004aa4553243 WatchSource:0}: Error finding container f1dce2859c686059393e355dca7c6d4f16e7c5ac938965ec13db004aa4553243: Status 404 returned error can't find the container with id f1dce2859c686059393e355dca7c6d4f16e7c5ac938965ec13db004aa4553243
Oct 30 12:40:42.057792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:42.057764    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"]
Oct 30 12:40:42.255965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:42.255922    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2" event={"ID":"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9","Type":"ContainerStarted","Data":"f1dce2859c686059393e355dca7c6d4f16e7c5ac938965ec13db004aa4553243"}
Oct 30 12:40:42.257777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:42.257748    5060 generic.go:356] "Generic (PLEG): container finished" podID="0e170462-cccd-4451-9970-58003dc1a40b" containerID="1cb5892259d0ea1ded4bee83c99e6041ea145da295625b35b1a2207e3f3e6efa" exitCode=0
Oct 30 12:40:42.257926 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:42.257793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885" event={"ID":"0e170462-cccd-4451-9970-58003dc1a40b","Type":"ContainerDied","Data":"1cb5892259d0ea1ded4bee83c99e6041ea145da295625b35b1a2207e3f3e6efa"}
Oct 30 12:40:42.808114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:42.808082    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:43.268200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.268122    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" containerID="bd014a24da78cd6e2a035a81da50a2cb0bb331308a26e20d77f5fbf21423ece5" exitCode=0
Oct 30 12:40:43.268356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.268208    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2" event={"ID":"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9","Type":"ContainerDied","Data":"bd014a24da78cd6e2a035a81da50a2cb0bb331308a26e20d77f5fbf21423ece5"}
Oct 30 12:40:43.410014 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.409974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:43.442689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.442666    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:43.522099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.522004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca\") pod \"0e170462-cccd-4451-9970-58003dc1a40b\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") "
Oct 30 12:40:43.522099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.522071    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca\") pod \"0e170462-cccd-4451-9970-58003dc1a40b\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") "
Oct 30 12:40:43.522328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.522193    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs\" (UniqueName: \"kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs\") pod \"0e170462-cccd-4451-9970-58003dc1a40b\" (UID: \"0e170462-cccd-4451-9970-58003dc1a40b\") "
Oct 30 12:40:43.525922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.524841    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca" pod "0e170462-cccd-4451-9970-58003dc1a40b" (UID: "0e170462-cccd-4451-9970-58003dc1a40b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:43.526371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.526342    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs" pod "0e170462-cccd-4451-9970-58003dc1a40b" (UID: "0e170462-cccd-4451-9970-58003dc1a40b"). InnerVolumeSpecName "kube-api-access-kkmbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:43.527566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.527438    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca" pod "0e170462-cccd-4451-9970-58003dc1a40b" (UID: "0e170462-cccd-4451-9970-58003dc1a40b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:43.624017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.623987    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:43.624017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.624016    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e170462-cccd-4451-9970-58003dc1a40b-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:43.624218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:43.624026    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmbs\" (UniqueName: \"kubernetes.io/projected/0e170462-cccd-4451-9970-58003dc1a40b-kube-api-access-kkmbs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:44.030580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.030543    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"]
Oct 30 12:40:44.037079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.037049    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318ab2b-ktvzs"]
Oct 30 12:40:44.278058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.278025    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"
Oct 30 12:40:44.278382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.278050    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885" event={"ID":"0e170462-cccd-4451-9970-58003dc1a40b","Type":"ContainerDied","Data":"e4c117a76d360e703871bd51079fd68ef5a34594922e88c7d40b1e09bc793300"}
Oct 30 12:40:44.278382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.278107    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c117a76d360e703871bd51079fd68ef5a34594922e88c7d40b1e09bc793300"
Oct 30 12:40:44.336069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.336041    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b291a3-5569-4ce5-bc03-85d0e15e9893" path="/var/lib/kubelet/pods/31b291a3-5569-4ce5-bc03-85d0e15e9893/volumes"
Oct 30 12:40:44.451248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.451224    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:44.538884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.538846    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca\") pod \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") "
Oct 30 12:40:44.539088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.538971    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca\") pod \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") "
Oct 30 12:40:44.539088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.539008    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq\" (UniqueName: \"kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq\") pod \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\" (UID: \"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9\") "
Oct 30 12:40:44.539344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.539285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca" pod "1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" (UID: "1f2fc0a2-cbfe-49ed-8941-89f04f7525c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:44.539801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.539772    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:44.541865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.541835    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca" pod "1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" (UID: "1f2fc0a2-cbfe-49ed-8941-89f04f7525c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:44.542620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.542600    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq" pod "1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" (UID: "1f2fc0a2-cbfe-49ed-8941-89f04f7525c9"). InnerVolumeSpecName "kube-api-access-nbmkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:44.641392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.641355    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:44.641392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.641391    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbmkq\" (UniqueName: \"kubernetes.io/projected/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9-kube-api-access-nbmkq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:44.690476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.681190    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:44.904841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:44.904812    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:40:45.046427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046390    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046500    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046540    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046611    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046652    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046726    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046787    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046815    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046911    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:45.046950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.046930    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo\") pod \"5689375b-457e-44eb-b779-75c5dd8d35fc\" (UID: \"5689375b-457e-44eb-b779-75c5dd8d35fc\") "
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047000    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047304    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047494    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047518    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047534    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047547    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.047617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.047560    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5689375b-457e-44eb-b779-75c5dd8d35fc-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.049295 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.049269    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "kube-api-access-bh5z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:45.049573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.049548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:45.049685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.049580    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:45.049933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.049912    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo" pod "5689375b-457e-44eb-b779-75c5dd8d35fc" (UID: "5689375b-457e-44eb-b779-75c5dd8d35fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:40:45.148488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.148399    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.148488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.148425    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.148488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.148435    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5z4\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-kube-api-access-bh5z4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.148488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.148446    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5689375b-457e-44eb-b779-75c5dd8d35fc-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:40:45.289070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.289037    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"
Oct 30 12:40:45.289255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.289066    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2" event={"ID":"1f2fc0a2-cbfe-49ed-8941-89f04f7525c9","Type":"ContainerDied","Data":"f1dce2859c686059393e355dca7c6d4f16e7c5ac938965ec13db004aa4553243"}
Oct 30 12:40:45.289255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.289118    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1dce2859c686059393e355dca7c6d4f16e7c5ac938965ec13db004aa4553243"
Oct 30 12:40:45.291368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.291341    5060 generic.go:356] "Generic (PLEG): container finished" podID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerID="0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231" exitCode=0
Oct 30 12:40:45.291550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.291394    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"
Oct 30 12:40:45.291550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.291428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" event={"ID":"5689375b-457e-44eb-b779-75c5dd8d35fc","Type":"ContainerDied","Data":"0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231"}
Oct 30 12:40:45.291550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.291515    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck" event={"ID":"5689375b-457e-44eb-b779-75c5dd8d35fc","Type":"ContainerDied","Data":"4910ec919c2c7a655f39438ba1c70027c9b0f542e19128865852bdc83359b446"}
Oct 30 12:40:45.291550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.291547    5060 scope.go:122] "RemoveContainer" containerID="0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231"
Oct 30 12:40:45.306485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.306428    5060 scope.go:122] "RemoveContainer" containerID="0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"
Oct 30 12:40:45.317488 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.317465    5060 scope.go:122] "RemoveContainer" containerID="0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231"
Oct 30 12:40:45.317780 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:45.317759    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231\": container with ID starting with 0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231 not found: ID does not exist" containerID="0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231"
Oct 30 12:40:45.317878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.317792    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231"} err="failed to get container status \"0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231\": rpc error: code = NotFound desc = could not find container \"0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231\": container with ID starting with 0ad1f15bca50900a53793d8910330e4a61e50b14b3018571490552e693de0231 not found: ID does not exist"
Oct 30 12:40:45.317878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.317816    5060 scope.go:122] "RemoveContainer" containerID="0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"
Oct 30 12:40:45.318084 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:45.318066    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e\": container with ID starting with 0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e not found: ID does not exist" containerID="0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"
Oct 30 12:40:45.318161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.318093    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e"} err="failed to get container status \"0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e\": rpc error: code = NotFound desc = could not find container \"0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e\": container with ID starting with 0df8b736ef96b271456a41d1f93b1432c251f244e2f016fa0e1f37d74db0dc3e not found: ID does not exist"
Oct 30 12:40:45.368492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.368420    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"]
Oct 30 12:40:45.380051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:45.380027    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-96wck"]
Oct 30 12:40:46.209029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.208991    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"]
Oct 30 12:40:46.210677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210649    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy"
Oct 30 12:40:46.210677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210676    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy"
Oct 30 12:40:46.210918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210764    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0e170462-cccd-4451-9970-58003dc1a40b" containerName="revoke-keys"
Oct 30 12:40:46.210918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210773    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e170462-cccd-4451-9970-58003dc1a40b" containerName="revoke-keys"
Oct 30 12:40:46.210918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210788    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" containerName="revoke-keys"
Oct 30 12:40:46.210918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210795    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" containerName="revoke-keys"
Oct 30 12:40:46.210918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210808    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy"
Oct 30 12:40:46.210918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.210815    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy"
Oct 30 12:40:46.211282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.211014    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e170462-cccd-4451-9970-58003dc1a40b" containerName="revoke-keys"
Oct 30 12:40:46.211282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.211027    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" containerName="revoke-keys"
Oct 30 12:40:46.211282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.211036    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy"
Oct 30 12:40:46.211282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.211050    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" containerName="istio-proxy"
Oct 30 12:40:46.218688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.218664    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.233589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.233561    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"]
Oct 30 12:40:46.238232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.238207    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0c1869\"/\"openshift-service-ca.crt\""
Oct 30 12:40:46.238363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.238297    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0c1869\"/\"kube-root-ca.crt\""
Oct 30 12:40:46.246282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.246261    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0c1869\"/\"default-dockercfg-kgczp\""
Oct 30 12:40:46.246399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.246298    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0c1869\"/\"rate-test-model-0c1869-kserve-self-signed-certs\""
Oct 30 12:40:46.296163 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.296123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:46.337146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.337105    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5689375b-457e-44eb-b779-75c5dd8d35fc" path="/var/lib/kubelet/pods/5689375b-457e-44eb-b779-75c5dd8d35fc/volumes"
Oct 30 12:40:46.363339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.363304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.363513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.363423    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.363569 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.363553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.363653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.363631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.363709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.363692    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.363754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.363737    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrtk\" (UniqueName: \"kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.466535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.466408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.466535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.466518    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.466803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.466610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.466803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.466662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.466803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.466752    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.466803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.466796    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrtk\" (UniqueName: \"kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.488670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.468472    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.488670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.468736    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.488670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.468961    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.515475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.512587    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.533484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.523509    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.546532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.540160    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:46.548205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.548133    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrtk\" (UniqueName: \"kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk\") pod \"rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") " pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:46.838190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:46.838101    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:47.151148 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:40:47.151112    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d3ce9be_cce5_466e_b3e9_f28e475fb26c.slice/crio-ea2f76d78291a62f3316f606b9b9b3a4117e02c59766ae3c7cf5ea79011e15f6 WatchSource:0}: Error finding container ea2f76d78291a62f3316f606b9b9b3a4117e02c59766ae3c7cf5ea79011e15f6: Status 404 returned error can't find the container with id ea2f76d78291a62f3316f606b9b9b3a4117e02c59766ae3c7cf5ea79011e15f6
Oct 30 12:40:47.152562 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:47.152540    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"]
Oct 30 12:40:47.319524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:47.319487    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" event={"ID":"9d3ce9be-cce5-466e-b3e9-f28e475fb26c","Type":"ContainerStarted","Data":"7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100"}
Oct 30 12:40:47.320010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:47.319536    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" event={"ID":"9d3ce9be-cce5-466e-b3e9-f28e475fb26c","Type":"ContainerStarted","Data":"ea2f76d78291a62f3316f606b9b9b3a4117e02c59766ae3c7cf5ea79011e15f6"}
Oct 30 12:40:47.366214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:47.366165    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" podStartSLOduration=1.366152523 podStartE2EDuration="1.366152523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:47.364217903 +0000 UTC m=+4251.159102397" watchObservedRunningTime="2026-10-30 12:40:47.366152523 +0000 UTC m=+4251.161037017"
Oct 30 12:40:47.839222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:47.839191    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:48.542891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:48.542861    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:48.866564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:48.866531    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:49.069439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:49.069396    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:49.861208 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:40:49.861168    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:40:50.122719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:50.122623    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:50.353235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:50.353196    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:51.145025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:51.144995    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:51.344294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:51.344254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:52.132231 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:52.132195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:53.889389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:53.889352    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:55.798181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:55.798146    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:56.677529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:56.677491    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:56.838624 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:56.838594    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:56.851601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:56.851579    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:57.385263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:57.385219    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:57.416534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:57.416507    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:40:57.656036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:57.655944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:59.443575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:40:59.443538    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:00.925622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:00.925578    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:01.444715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:01.444682    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:01.571856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:01.571821    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"]
Oct 30 12:41:01.586891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:01.586847    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b8a9d-7z885"]
Oct 30 12:41:01.637977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:01.637941    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:02.173670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:02.173628    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:02.337526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:02.337486    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e170462-cccd-4451-9970-58003dc1a40b" path="/var/lib/kubelet/pods/0e170462-cccd-4451-9970-58003dc1a40b/volumes"
Oct 30 12:41:03.365848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:03.365805    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:03.523257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:03.523221    5060 scope.go:122] "RemoveContainer" containerID="5f82ddbe09bad9040d8ec33f6234a8bb1ac8f87aaffc57eb425a36bdc5ae564e"
Oct 30 12:41:03.534677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:03.534656    5060 scope.go:122] "RemoveContainer" containerID="ae80d49d4a862e4533e1a261c3bda494aa64a58b962a49b12a6930c46470afa4"
Oct 30 12:41:03.545887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:03.545865    5060 scope.go:122] "RemoveContainer" containerID="73c7b2deee14689ed5365990240db85cad69b5b18991b7b2c72ab4851e156e0e"
Oct 30 12:41:04.939034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:04.939001    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:05.534302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:05.534270    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:05.658589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:05.658547    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"]
Oct 30 12:41:05.670580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:05.670554    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4e261b-2crvw"]
Oct 30 12:41:05.716825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:05.712345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:06.337660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:06.337579    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091da81a-7800-496f-9b0a-249595b44f32" path="/var/lib/kubelet/pods/091da81a-7800-496f-9b0a-249595b44f32/volumes"
Oct 30 12:41:06.381951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:06.381909    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:07.660344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:07.660290    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:09.195558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:09.195514    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:09.350780 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:09.350742    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:09.520621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:09.520521    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:09.676403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:09.676364    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:10.130747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.130715    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"]
Oct 30 12:41:10.135520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.135497    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.153045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.153023    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:41:10.153724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.153704    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:41:10.165795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.165776    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:41:10.165894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.165795    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:41:10.181483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.181440    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"]
Oct 30 12:41:10.222620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.222585    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.223076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.222633    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.223076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.222795    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdtb\" (UniqueName: \"kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.324875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.324835    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdtb\" (UniqueName: \"kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.325118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.325102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.325188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.325140    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.326059 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.325983    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.328203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.328174    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.348529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.348505    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdtb\" (UniqueName: \"kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.449932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.449909    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:10.761614 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:41:10.761564    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f3b2f2_c154_4326_b1eb_b19f39fc9704.slice/crio-5506a1aef4c0c62ead139779af9c9b0226570fc705e3e5ee70be4f581c87b487 WatchSource:0}: Error finding container 5506a1aef4c0c62ead139779af9c9b0226570fc705e3e5ee70be4f581c87b487: Status 404 returned error can't find the container with id 5506a1aef4c0c62ead139779af9c9b0226570fc705e3e5ee70be4f581c87b487
Oct 30 12:41:10.763891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:10.763869    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"]
Oct 30 12:41:11.536615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:11.536581    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:11.561246 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:11.561206    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf" event={"ID":"19f3b2f2-c154-4326-b1eb-b19f39fc9704","Type":"ContainerStarted","Data":"5bd7b7086fd624cfa270c9fc13c460eaf5fef6090329018061379c690b20df1f"}
Oct 30 12:41:11.561671 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:11.561645    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf" event={"ID":"19f3b2f2-c154-4326-b1eb-b19f39fc9704","Type":"ContainerStarted","Data":"5506a1aef4c0c62ead139779af9c9b0226570fc705e3e5ee70be4f581c87b487"}
Oct 30 12:41:11.624249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:11.624196    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf" podStartSLOduration=1.6241810220000001 podStartE2EDuration="1.624181022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:11.623186445 +0000 UTC m=+4275.418070937" watchObservedRunningTime="2026-10-30 12:41:11.624181022 +0000 UTC m=+4275.419065562"
Oct 30 12:41:12.571733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:12.571692    5060 generic.go:356] "Generic (PLEG): container finished" podID="19f3b2f2-c154-4326-b1eb-b19f39fc9704" containerID="5bd7b7086fd624cfa270c9fc13c460eaf5fef6090329018061379c690b20df1f" exitCode=0
Oct 30 12:41:12.572145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:12.571771    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf" event={"ID":"19f3b2f2-c154-4326-b1eb-b19f39fc9704","Type":"ContainerDied","Data":"5bd7b7086fd624cfa270c9fc13c460eaf5fef6090329018061379c690b20df1f"}
Oct 30 12:41:13.039895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.039856    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:13.276091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.276057    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:13.767145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.767118    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:13.881208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.881119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca\") pod \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") "
Oct 30 12:41:13.881377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.881300    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca\") pod \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") "
Oct 30 12:41:13.881492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.881402    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb\" (UniqueName: \"kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb\") pod \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\" (UID: \"19f3b2f2-c154-4326-b1eb-b19f39fc9704\") "
Oct 30 12:41:13.881757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.881734    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca" pod "19f3b2f2-c154-4326-b1eb-b19f39fc9704" (UID: "19f3b2f2-c154-4326-b1eb-b19f39fc9704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:13.882306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.882282    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:13.883666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.883633    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca" pod "19f3b2f2-c154-4326-b1eb-b19f39fc9704" (UID: "19f3b2f2-c154-4326-b1eb-b19f39fc9704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:41:13.884194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.884166    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb" pod "19f3b2f2-c154-4326-b1eb-b19f39fc9704" (UID: "19f3b2f2-c154-4326-b1eb-b19f39fc9704"). InnerVolumeSpecName "kube-api-access-lvdtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:13.983197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.983160    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdtb\" (UniqueName: \"kubernetes.io/projected/19f3b2f2-c154-4326-b1eb-b19f39fc9704-kube-api-access-lvdtb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:13.983197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:13.983195    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f3b2f2-c154-4326-b1eb-b19f39fc9704-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:14.172662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.172575    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:14.328972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.328930    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:14.531274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.531237    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:14.665808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.665769    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf" event={"ID":"19f3b2f2-c154-4326-b1eb-b19f39fc9704","Type":"ContainerDied","Data":"5506a1aef4c0c62ead139779af9c9b0226570fc705e3e5ee70be4f581c87b487"}
Oct 30 12:41:14.666309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.666287    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5506a1aef4c0c62ead139779af9c9b0226570fc705e3e5ee70be4f581c87b487"
Oct 30 12:41:14.666445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.666231    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"
Oct 30 12:41:14.855978 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:14.855883    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:15.018878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:15.018837    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:15.198717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:15.198674    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:17.211661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:17.211627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:18.956263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:18.956227    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:19.858252 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:41:19.858214    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:41:20.745577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:20.745543    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:20.862235 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:20.862202    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"]
Oct 30 12:41:20.894413 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:20.894381    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"]
Oct 30 12:41:21.036733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:21.036655    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:21.177920 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:21.177888    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:22.489094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:22.489057    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:24.258501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:24.258435    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:25.863396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:25.863324    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c" podUID="0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" containerName="payload-processing" containerID="cri-o://f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964" gracePeriod=25
Oct 30 12:41:25.895380 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:25.895337    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78" podUID="1ca546bf-94a7-4db4-81fe-c3208ba6223f" containerName="payload-pre-processing" containerID="cri-o://abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63" gracePeriod=25
Oct 30 12:41:25.971427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:25.971392    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:26.202078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.202051    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:41:26.222845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.222817    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:41:26.295910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.295877    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf\" (UniqueName: \"kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf\") pod \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") "
Oct 30 12:41:26.297020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.296992    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp\" (UniqueName: \"kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp\") pod \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") "
Oct 30 12:41:26.297518 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.297488    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume\") pod \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\" (UID: \"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e\") "
Oct 30 12:41:26.298128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.298020    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume" pod "0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" (UID: "0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:26.298334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.298317    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume\") pod \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\" (UID: \"1ca546bf-94a7-4db4-81fe-c3208ba6223f\") "
Oct 30 12:41:26.299403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.298917    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume" pod "1ca546bf-94a7-4db4-81fe-c3208ba6223f" (UID: "1ca546bf-94a7-4db4-81fe-c3208ba6223f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:26.299403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.299029    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf" pod "0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" (UID: "0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e"). InnerVolumeSpecName "kube-api-access-6rgbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:26.299403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.299364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp" pod "1ca546bf-94a7-4db4-81fe-c3208ba6223f" (UID: "1ca546bf-94a7-4db4-81fe-c3208ba6223f"). InnerVolumeSpecName "kube-api-access-tbwtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:26.299693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.299587    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:26.299693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.299600    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca546bf-94a7-4db4-81fe-c3208ba6223f-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:26.299693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.299609    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rgbf\" (UniqueName: \"kubernetes.io/projected/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e-kube-api-access-6rgbf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:26.299693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.299618    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwtp\" (UniqueName: \"kubernetes.io/projected/1ca546bf-94a7-4db4-81fe-c3208ba6223f-kube-api-access-tbwtp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:26.782692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.782657    5060 generic.go:356] "Generic (PLEG): container finished" podID="1ca546bf-94a7-4db4-81fe-c3208ba6223f" containerID="abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63" exitCode=0
Oct 30 12:41:26.782922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.782719    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"
Oct 30 12:41:26.782922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.782743    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78" event={"ID":"1ca546bf-94a7-4db4-81fe-c3208ba6223f","Type":"ContainerDied","Data":"abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63"}
Oct 30 12:41:26.782922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.782800    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78" event={"ID":"1ca546bf-94a7-4db4-81fe-c3208ba6223f","Type":"ContainerDied","Data":"76cb793ccfdd02c24c29b224f86085585885f98a106da164b2513ea555f8935d"}
Oct 30 12:41:26.782922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.782832    5060 scope.go:122] "RemoveContainer" containerID="abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63"
Oct 30 12:41:26.784695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.784671    5060 generic.go:356] "Generic (PLEG): container finished" podID="0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" containerID="f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964" exitCode=0
Oct 30 12:41:26.784831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.784764    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c" event={"ID":"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e","Type":"ContainerDied","Data":"f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964"}
Oct 30 12:41:26.784831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.784807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c" event={"ID":"0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e","Type":"ContainerDied","Data":"0ae6eea3f3f1e14e18fb120a3fdc3fd8ffbfa07b63dfbc9cc20001470134c06d"}
Oct 30 12:41:26.784975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.784874    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"
Oct 30 12:41:26.798193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.798164    5060 scope.go:122] "RemoveContainer" containerID="abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63"
Oct 30 12:41:26.798592 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:41:26.798562    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63\": container with ID starting with abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63 not found: ID does not exist" containerID="abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63"
Oct 30 12:41:26.798719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.798602    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63"} err="failed to get container status \"abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63\": rpc error: code = NotFound desc = could not find container \"abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63\": container with ID starting with abe2079e2247820ecfb1e6bc15c417b7bb59f806bb4b7ce0597d96fc8c4b5b63 not found: ID does not exist"
Oct 30 12:41:26.798719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.798628    5060 scope.go:122] "RemoveContainer" containerID="f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964"
Oct 30 12:41:26.812097 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.812076    5060 scope.go:122] "RemoveContainer" containerID="f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964"
Oct 30 12:41:26.812405 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:41:26.812377    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964\": container with ID starting with f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964 not found: ID does not exist" containerID="f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964"
Oct 30 12:41:26.812524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.812408    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964"} err="failed to get container status \"f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964\": rpc error: code = NotFound desc = could not find container \"f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964\": container with ID starting with f69c27b5f0887c300c158610d9597f2202068e265248bb78e4bcbae9c379a964 not found: ID does not exist"
Oct 30 12:41:26.839647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.839604    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"]
Oct 30 12:41:26.851976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.851953    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-28ca05-d8ff875bc-dpg78"]
Oct 30 12:41:26.880443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.880413    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"]
Oct 30 12:41:26.886321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:26.886294    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-28ca05-5c69bfcf9d-pjj5c"]
Oct 30 12:41:27.697066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:27.697028    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:28.334927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:28.334894    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" path="/var/lib/kubelet/pods/0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e/volumes"
Oct 30 12:41:28.335388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:28.335370    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca546bf-94a7-4db4-81fe-c3208ba6223f" path="/var/lib/kubelet/pods/1ca546bf-94a7-4db4-81fe-c3208ba6223f/volumes"
Oct 30 12:41:29.422874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:29.422835    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:31.214057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:31.214021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:32.940902 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:32.940863    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:34.693057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:34.693021    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:36.148635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:36.148592    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:36.741635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:36.741594    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:36.849751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:36.849716    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"]
Oct 30 12:41:36.859823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:36.859799    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-28ca05-p5ldf"]
Oct 30 12:41:36.902389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:36.902345    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:37.374687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:37.374654    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:38.336364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:38.336324    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f3b2f2-c154-4326-b1eb-b19f39fc9704" path="/var/lib/kubelet/pods/19f3b2f2-c154-4326-b1eb-b19f39fc9704/volumes"
Oct 30 12:41:38.745795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:38.745759    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:40.426664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:40.426620    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:42.119027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:42.118725    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:43.843735 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:43.843698    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:43.988240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:43.988201    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:44.193249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.193146    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:44.762955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.762920    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"]
Oct 30 12:41:44.763545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763522    5060 cpu_manager.go:414] "containerMap: removing container" podUID="19f3b2f2-c154-4326-b1eb-b19f39fc9704" containerName="revoke-keys"
Oct 30 12:41:44.763545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763542    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f3b2f2-c154-4326-b1eb-b19f39fc9704" containerName="revoke-keys"
Oct 30 12:41:44.763830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763578    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1ca546bf-94a7-4db4-81fe-c3208ba6223f" containerName="payload-pre-processing"
Oct 30 12:41:44.763830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763587    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca546bf-94a7-4db4-81fe-c3208ba6223f" containerName="payload-pre-processing"
Oct 30 12:41:44.763830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763599    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" containerName="payload-processing"
Oct 30 12:41:44.763830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763606    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" containerName="payload-processing"
Oct 30 12:41:44.763830 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763818    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb8e3d4-f91b-4820-9e40-9ccbfdca7b0e" containerName="payload-processing"
Oct 30 12:41:44.764128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763847    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f3b2f2-c154-4326-b1eb-b19f39fc9704" containerName="revoke-keys"
Oct 30 12:41:44.764128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.763862    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca546bf-94a7-4db4-81fe-c3208ba6223f" containerName="payload-pre-processing"
Oct 30 12:41:44.768764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.768743    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.787293 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.787266    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"]
Oct 30 12:41:44.794468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.794017    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:41:44.794468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.794061    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:41:44.794468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.794134    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:41:44.798465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.794759    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:41:44.868337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.868298    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.868337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.868339    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.868812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.868375    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7d6\" (UniqueName: \"kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.970256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.970222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.970256 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.970259    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.970550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.970296    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7d6\" (UniqueName: \"kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.971000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.970974    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.972875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.972850    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:44.994153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:44.994127    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7d6\" (UniqueName: \"kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:45.083997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:45.083915    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:45.393584 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:41:45.393549    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2b9282_181b_40ad_85fd_f4f195b7c5fc.slice/crio-e25e5aa8e3217fceade89e9340b965d9523506b48c7b93256680288a516e529d WatchSource:0}: Error finding container e25e5aa8e3217fceade89e9340b965d9523506b48c7b93256680288a516e529d: Status 404 returned error can't find the container with id e25e5aa8e3217fceade89e9340b965d9523506b48c7b93256680288a516e529d
Oct 30 12:41:45.394268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:45.394244    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"]
Oct 30 12:41:45.762585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:45.762545    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:45.967130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:45.967090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9" event={"ID":"8b2b9282-181b-40ad-85fd-f4f195b7c5fc","Type":"ContainerStarted","Data":"44c1b1d0364182af715e6dc7cedffbb8635d4dfb49b39fbc2948028b2d7058bc"}
Oct 30 12:41:45.967675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:45.967174    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9" event={"ID":"8b2b9282-181b-40ad-85fd-f4f195b7c5fc","Type":"ContainerStarted","Data":"e25e5aa8e3217fceade89e9340b965d9523506b48c7b93256680288a516e529d"}
Oct 30 12:41:46.011406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:46.011359    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9" podStartSLOduration=2.011344701 podStartE2EDuration="2.011344701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:46.008137251 +0000 UTC m=+4309.803021745" watchObservedRunningTime="2026-10-30 12:41:46.011344701 +0000 UTC m=+4309.806229217"
Oct 30 12:41:46.977347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:46.977307    5060 generic.go:356] "Generic (PLEG): container finished" podID="8b2b9282-181b-40ad-85fd-f4f195b7c5fc" containerID="44c1b1d0364182af715e6dc7cedffbb8635d4dfb49b39fbc2948028b2d7058bc" exitCode=0
Oct 30 12:41:46.977873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:46.977423    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9" event={"ID":"8b2b9282-181b-40ad-85fd-f4f195b7c5fc","Type":"ContainerDied","Data":"44c1b1d0364182af715e6dc7cedffbb8635d4dfb49b39fbc2948028b2d7058bc"}
Oct 30 12:41:47.560769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:47.560734    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:48.155466 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.155429    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:48.219355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.219318    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca\") pod \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") "
Oct 30 12:41:48.219597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.219577    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6\" (UniqueName: \"kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6\") pod \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") "
Oct 30 12:41:48.219689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.219637    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca\") pod \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\" (UID: \"8b2b9282-181b-40ad-85fd-f4f195b7c5fc\") "
Oct 30 12:41:48.219817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.219797    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca" pod "8b2b9282-181b-40ad-85fd-f4f195b7c5fc" (UID: "8b2b9282-181b-40ad-85fd-f4f195b7c5fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:48.220244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.220219    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:48.222597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.222571    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca" pod "8b2b9282-181b-40ad-85fd-f4f195b7c5fc" (UID: "8b2b9282-181b-40ad-85fd-f4f195b7c5fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:41:48.222919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.222865    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6" pod "8b2b9282-181b-40ad-85fd-f4f195b7c5fc" (UID: "8b2b9282-181b-40ad-85fd-f4f195b7c5fc"). InnerVolumeSpecName "kube-api-access-ll7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:48.322362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.322270    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7d6\" (UniqueName: \"kubernetes.io/projected/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-kube-api-access-ll7d6\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:48.322362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.322300    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b9282-181b-40ad-85fd-f4f195b7c5fc-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:41:48.998619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.998592    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"
Oct 30 12:41:48.998846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.998635    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9" event={"ID":"8b2b9282-181b-40ad-85fd-f4f195b7c5fc","Type":"ContainerDied","Data":"e25e5aa8e3217fceade89e9340b965d9523506b48c7b93256680288a516e529d"}
Oct 30 12:41:48.998846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:48.998690    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25e5aa8e3217fceade89e9340b965d9523506b48c7b93256680288a516e529d"
Oct 30 12:41:49.344474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:49.344421    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:49.858684 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:41:49.858645    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:41:51.250225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:51.250188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:53.035102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:53.035068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:54.706436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:54.706396    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:55.389842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:55.389806    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:55.527287 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:55.527251    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:56.438248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:56.438207    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:56.799661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:56.799628    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:58.168934 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:58.168887    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:59.836003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:41:59.835965    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:01.554119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:01.554070    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:03.281433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:03.281393    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:03.614846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:03.614759    5060 scope.go:122] "RemoveContainer" containerID="0ddc0ad463ed30e86aead1487d19ced67cff18d25c747f9210ef2f184499ee02"
Oct 30 12:42:03.629349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:03.629326    5060 scope.go:122] "RemoveContainer" containerID="1542ebcd4d41851bdf107f941e88b7b2346ef7f6128714f090e10175bc7015ea"
Oct 30 12:42:03.641901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:03.641880    5060 scope.go:122] "RemoveContainer" containerID="55fb46b87eae2c6fbc74b389cb8efe8d4d3697f94c6aa7f91f83fec3718a7eda"
Oct 30 12:42:03.653676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:03.653654    5060 scope.go:122] "RemoveContainer" containerID="31542232f5fed8a42d876fd250feef789eafedc04f482f19840ce55d89a21907"
Oct 30 12:42:03.664316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:03.664294    5060 scope.go:122] "RemoveContainer" containerID="2f135f0dbecaccabf4fc449dcba4f211531355bbbe320d998fac147011c652b2"
Oct 30 12:42:05.162873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:05.162838    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:06.961336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:06.961296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:08.031142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:08.031105    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"]
Oct 30 12:42:08.043186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:08.043160    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a403d2d8-xdmh7"]
Oct 30 12:42:08.335604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:08.335516    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab40780-0645-40c8-a142-ad820778924f" path="/var/lib/kubelet/pods/bab40780-0645-40c8-a142-ad820778924f/volumes"
Oct 30 12:42:08.646351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:08.646254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:10.312571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:10.312517    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:11.994401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:11.994363    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:13.744117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:13.744086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:15.462760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:15.462722    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:16.976744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:16.976703    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:17.561099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:17.561064    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:17.676058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:17.676024    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"]
Oct 30 12:42:17.681871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:17.681844    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b02580-rvhw9"]
Oct 30 12:42:17.729326 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:17.729282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:17.916169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:17.916046    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:18.248845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:18.248799    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:18.336241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:18.336203    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2b9282-181b-40ad-85fd-f4f195b7c5fc" path="/var/lib/kubelet/pods/8b2b9282-181b-40ad-85fd-f4f195b7c5fc/volumes"
Oct 30 12:42:19.745087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:19.745040    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:19.858070 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:19.858028    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:42:21.546839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:21.546803    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:23.273550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:23.273510    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:24.973520 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:24.973482    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:25.139176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:25.139143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:25.274859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:25.274316    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"]
Oct 30 12:42:25.275685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:25.275360    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy" containerID="cri-o://007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53" gracePeriod=30
Oct 30 12:42:25.358227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:25.358188    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:25.817139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:25.817091    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:26.047859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.047824    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"]
Oct 30 12:42:26.048436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.048422    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8b2b9282-181b-40ad-85fd-f4f195b7c5fc" containerName="revoke-keys"
Oct 30 12:42:26.048507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.048437    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2b9282-181b-40ad-85fd-f4f195b7c5fc" containerName="revoke-keys"
Oct 30 12:42:26.048593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.048581    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2b9282-181b-40ad-85fd-f4f195b7c5fc" containerName="revoke-keys"
Oct 30 12:42:26.053500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.053448    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.069408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.069353    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:42:26.069558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.069490    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:42:26.070130 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.070107    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:42:26.070273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.070166    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:42:26.070493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.070436    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"]
Oct 30 12:42:26.133841 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.133798    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqv5\" (UniqueName: \"kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.134034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.133949    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.134034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.134019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.234965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.234931    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqv5\" (UniqueName: \"kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.235167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.235007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.235167 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.235053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.235694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.235669    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.237663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.237640    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.259370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.259341    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqv5\" (UniqueName: \"kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5\") pod \"maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.372815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.372743    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:26.679257 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:42:26.679222    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2149452_c5c6_47ff_8700_d304d4a39026.slice/crio-09e596e3e9b4a06369c6865b5307e6a5f791f5562eb5a92b5c27794d39f26d98 WatchSource:0}: Error finding container 09e596e3e9b4a06369c6865b5307e6a5f791f5562eb5a92b5c27794d39f26d98: Status 404 returned error can't find the container with id 09e596e3e9b4a06369c6865b5307e6a5f791f5562eb5a92b5c27794d39f26d98
Oct 30 12:42:26.679649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.679623    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"]
Oct 30 12:42:26.798071 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.798036    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:26.933107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:26.933023    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:27.341773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:27.341732    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh" event={"ID":"f2149452-c5c6-47ff-8700-d304d4a39026","Type":"ContainerStarted","Data":"dec2bb153bf0cb2049861bb8c3cd0d1462056343004043034e524bb470969c26"}
Oct 30 12:42:27.342219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:27.341793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh" event={"ID":"f2149452-c5c6-47ff-8700-d304d4a39026","Type":"ContainerStarted","Data":"09e596e3e9b4a06369c6865b5307e6a5f791f5562eb5a92b5c27794d39f26d98"}
Oct 30 12:42:27.384109 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:27.384058    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh" podStartSLOduration=1.384044993 podStartE2EDuration="1.384044993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:27.381304687 +0000 UTC m=+4351.176189180" watchObservedRunningTime="2026-10-30 12:42:27.384044993 +0000 UTC m=+4351.178929548"
Oct 30 12:42:28.350113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:28.350079    5060 generic.go:356] "Generic (PLEG): container finished" podID="f2149452-c5c6-47ff-8700-d304d4a39026" containerID="dec2bb153bf0cb2049861bb8c3cd0d1462056343004043034e524bb470969c26" exitCode=0
Oct 30 12:42:28.350560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:28.350164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh" event={"ID":"f2149452-c5c6-47ff-8700-d304d4a39026","Type":"ContainerDied","Data":"dec2bb153bf0cb2049861bb8c3cd0d1462056343004043034e524bb470969c26"}
Oct 30 12:42:28.568641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:28.568598    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:29.539501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.539473    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:29.577047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.577016    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca\") pod \"f2149452-c5c6-47ff-8700-d304d4a39026\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") "
Oct 30 12:42:29.577204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.577070    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca\") pod \"f2149452-c5c6-47ff-8700-d304d4a39026\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") "
Oct 30 12:42:29.577260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.577245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5\" (UniqueName: \"kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5\") pod \"f2149452-c5c6-47ff-8700-d304d4a39026\" (UID: \"f2149452-c5c6-47ff-8700-d304d4a39026\") "
Oct 30 12:42:29.577443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.577424    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca" pod "f2149452-c5c6-47ff-8700-d304d4a39026" (UID: "f2149452-c5c6-47ff-8700-d304d4a39026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:42:29.577839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.577818    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:29.582025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.581996    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5" pod "f2149452-c5c6-47ff-8700-d304d4a39026" (UID: "f2149452-c5c6-47ff-8700-d304d4a39026"). InnerVolumeSpecName "kube-api-access-fjqv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:29.585538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.585511    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca" pod "f2149452-c5c6-47ff-8700-d304d4a39026" (UID: "f2149452-c5c6-47ff-8700-d304d4a39026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:29.678901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.678823    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2149452-c5c6-47ff-8700-d304d4a39026-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:29.678901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:29.678855    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqv5\" (UniqueName: \"kubernetes.io/projected/f2149452-c5c6-47ff-8700-d304d4a39026-kube-api-access-fjqv5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.229681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.229640    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:30.366312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.366278    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"
Oct 30 12:42:30.366491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.366316    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh" event={"ID":"f2149452-c5c6-47ff-8700-d304d4a39026","Type":"ContainerDied","Data":"09e596e3e9b4a06369c6865b5307e6a5f791f5562eb5a92b5c27794d39f26d98"}
Oct 30 12:42:30.366491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.366362    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e596e3e9b4a06369c6865b5307e6a5f791f5562eb5a92b5c27794d39f26d98"
Oct 30 12:42:30.556196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.556172    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:42:30.591124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591092    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591187    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591211    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591265    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591322    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591375    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591489    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591539    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:30.591773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591546    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data\") pod \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\" (UID: \"9a37c5b7-b5a1-4389-baa2-692562ae1b96\") "
Oct 30 12:42:30.591773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591736    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:30.592128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591770    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:30.592128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.591956    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:30.592259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.592192    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.592259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.592213    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.592259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.592228    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.592259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.592240    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.592259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.592212    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:42:30.593815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.593787    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:42:30.594180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.594161    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:30.594392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.594372    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:30.594502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.594409    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42" pod "9a37c5b7-b5a1-4389-baa2-692562ae1b96" (UID: "9a37c5b7-b5a1-4389-baa2-692562ae1b96"). InnerVolumeSpecName "kube-api-access-lzj42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:30.693157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.693123    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.693157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.693150    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.693157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.693162    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzj42\" (UniqueName: \"kubernetes.io/projected/9a37c5b7-b5a1-4389-baa2-692562ae1b96-kube-api-access-lzj42\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.693407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.693173    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.693407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.693184    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a37c5b7-b5a1-4389-baa2-692562ae1b96-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:30.740721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.740685    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"]
Oct 30 12:42:30.754543 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.754497    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"]
Oct 30 12:42:30.765625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.765597    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"]
Oct 30 12:42:30.898647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:30.898530    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:31.377342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.377306    5060 generic.go:356] "Generic (PLEG): container finished" podID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerID="007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53" exitCode=0
Oct 30 12:42:31.377560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.377366    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"
Oct 30 12:42:31.377560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.377391    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" event={"ID":"9a37c5b7-b5a1-4389-baa2-692562ae1b96","Type":"ContainerDied","Data":"007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53"}
Oct 30 12:42:31.377560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.377445    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft" event={"ID":"9a37c5b7-b5a1-4389-baa2-692562ae1b96","Type":"ContainerDied","Data":"793e344d32d16f42cbdb8937fe831e730627e3b858710a49004fa4bb2e750d8a"}
Oct 30 12:42:31.377560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.377497    5060 scope.go:122] "RemoveContainer" containerID="007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53"
Oct 30 12:42:31.391733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.391710    5060 scope.go:122] "RemoveContainer" containerID="d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"
Oct 30 12:42:31.402816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.402604    5060 scope.go:122] "RemoveContainer" containerID="007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53"
Oct 30 12:42:31.402941 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:31.402921    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53\": container with ID starting with 007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53 not found: ID does not exist" containerID="007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53"
Oct 30 12:42:31.402988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.402950    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53"} err="failed to get container status \"007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53\": rpc error: code = NotFound desc = could not find container \"007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53\": container with ID starting with 007203b09608e23b741bc9d6183e6635acc4045da5cd47dbd3542224a0724c53 not found: ID does not exist"
Oct 30 12:42:31.402988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.402967    5060 scope.go:122] "RemoveContainer" containerID="d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"
Oct 30 12:42:31.403256 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:31.403229    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8\": container with ID starting with d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8 not found: ID does not exist" containerID="d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"
Oct 30 12:42:31.403315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.403266    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8"} err="failed to get container status \"d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8\": rpc error: code = NotFound desc = could not find container \"d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8\": container with ID starting with d9f249d56ca49a957c7e604b4398eaf233531f81c5f7103f79b955b0904d71b8 not found: ID does not exist"
Oct 30 12:42:31.440652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.440627    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"]
Oct 30 12:42:31.453082 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:31.453056    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b02580-openshift-default-5b8db95499-ffbft"]
Oct 30 12:42:32.062228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:32.061904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:32.335191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:32.335112    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" path="/var/lib/kubelet/pods/9a37c5b7-b5a1-4389-baa2-692562ae1b96/volumes"
Oct 30 12:42:33.833302 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:33.833266    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:35.644507 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:35.644426    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:36.687600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:36.687502    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:36.839985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:36.839940    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:37.470611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:37.470572    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:39.123709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:39.123666    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:40.857966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:40.857924    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:42.641389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:42.641347    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:44.299074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:44.299032    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:45.764244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:45.764180    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main" containerID="cri-o://1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084" gracePeriod=45
Oct 30 12:42:45.778124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:45.778062    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" containerID="cri-o://9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463" gracePeriod=45
Oct 30 12:42:45.791313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:45.791277    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" containerID="cri-o://0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526" gracePeriod=45
Oct 30 12:42:45.799040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:45.799011    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": read tcp 10.132.0.2:36926->10.132.0.63:8000: read: connection reset by peer"
Oct 30 12:42:46.051385 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.051296    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:46.184729 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.184702    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 12:42:46.188505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.187936    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 12:42:46.196693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.194680    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 12:42:46.228540 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228506    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs\") pod \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") "
Oct 30 12:42:46.228750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm\") pod \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") "
Oct 30 12:42:46.228750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228630    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache\") pod \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") "
Oct 30 12:42:46.228750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228668    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir\") pod \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") "
Oct 30 12:42:46.228750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir\") pod \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") "
Oct 30 12:42:46.228992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228758    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw\" (UniqueName: \"kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw\") pod \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") "
Oct 30 12:42:46.228992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228804    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache\") pod \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") "
Oct 30 12:42:46.228992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228871    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs\" (UniqueName: \"kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs\") pod \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") "
Oct 30 12:42:46.228992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.228969    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm\") pod \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") "
Oct 30 12:42:46.229225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229015    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm\") pod \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") "
Oct 30 12:42:46.229225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229055    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home\") pod \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") "
Oct 30 12:42:46.229225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229091    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache\") pod \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") "
Oct 30 12:42:46.229399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229277    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir\") pod \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\" (UID: \"d12ab22d-2ee9-4c17-a3da-7001632db2fa\") "
Oct 30 12:42:46.229399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229339    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl\" (UniqueName: \"kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl\") pod \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") "
Oct 30 12:42:46.229536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229397    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg\" (UniqueName: \"kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg\") pod \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") "
Oct 30 12:42:46.229536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229475    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs\" (UniqueName: \"kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs\") pod \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") "
Oct 30 12:42:46.229536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229507    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home\") pod \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\" (UID: \"095a98a9-dc08-42d0-8ef6-5bdb796c2c54\") "
Oct 30 12:42:46.229695 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.229548    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home\") pod \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\" (UID: \"b84214b8-3b0f-4902-82f2-1cb36c3f2e99\") "
Oct 30 12:42:46.230401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.230374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home" pod "b84214b8-3b0f-4902-82f2-1cb36c3f2e99" (UID: "b84214b8-3b0f-4902-82f2-1cb36c3f2e99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.231638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.231091    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache" pod "095a98a9-dc08-42d0-8ef6-5bdb796c2c54" (UID: "095a98a9-dc08-42d0-8ef6-5bdb796c2c54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.231638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.231368    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir" pod "095a98a9-dc08-42d0-8ef6-5bdb796c2c54" (UID: "095a98a9-dc08-42d0-8ef6-5bdb796c2c54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.233410 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.233381    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir" pod "d12ab22d-2ee9-4c17-a3da-7001632db2fa" (UID: "d12ab22d-2ee9-4c17-a3da-7001632db2fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.233640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.233534    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs" pod "d12ab22d-2ee9-4c17-a3da-7001632db2fa" (UID: "d12ab22d-2ee9-4c17-a3da-7001632db2fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:46.237798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.237540    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs" pod "095a98a9-dc08-42d0-8ef6-5bdb796c2c54" (UID: "095a98a9-dc08-42d0-8ef6-5bdb796c2c54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:46.239470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.239428    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm" pod "d12ab22d-2ee9-4c17-a3da-7001632db2fa" (UID: "d12ab22d-2ee9-4c17-a3da-7001632db2fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.239470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.239439    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home" pod "095a98a9-dc08-42d0-8ef6-5bdb796c2c54" (UID: "095a98a9-dc08-42d0-8ef6-5bdb796c2c54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.239702 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.239683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache" pod "b84214b8-3b0f-4902-82f2-1cb36c3f2e99" (UID: "b84214b8-3b0f-4902-82f2-1cb36c3f2e99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.239877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.239858    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache" pod "d12ab22d-2ee9-4c17-a3da-7001632db2fa" (UID: "d12ab22d-2ee9-4c17-a3da-7001632db2fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.239961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.239870    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm" pod "095a98a9-dc08-42d0-8ef6-5bdb796c2c54" (UID: "095a98a9-dc08-42d0-8ef6-5bdb796c2c54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.240036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.240010    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home" pod "d12ab22d-2ee9-4c17-a3da-7001632db2fa" (UID: "d12ab22d-2ee9-4c17-a3da-7001632db2fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.240181 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.240057    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir" pod "b84214b8-3b0f-4902-82f2-1cb36c3f2e99" (UID: "b84214b8-3b0f-4902-82f2-1cb36c3f2e99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.240405 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.240380    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg" pod "b84214b8-3b0f-4902-82f2-1cb36c3f2e99" (UID: "b84214b8-3b0f-4902-82f2-1cb36c3f2e99"). InnerVolumeSpecName "kube-api-access-dc8xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:46.240991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.240816    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw" pod "d12ab22d-2ee9-4c17-a3da-7001632db2fa" (UID: "d12ab22d-2ee9-4c17-a3da-7001632db2fa"). InnerVolumeSpecName "kube-api-access-w7nsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:46.240991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.240958    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm" pod "b84214b8-3b0f-4902-82f2-1cb36c3f2e99" (UID: "b84214b8-3b0f-4902-82f2-1cb36c3f2e99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:46.241285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.241267    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl" pod "095a98a9-dc08-42d0-8ef6-5bdb796c2c54" (UID: "095a98a9-dc08-42d0-8ef6-5bdb796c2c54"). InnerVolumeSpecName "kube-api-access-6htbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:46.242660 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.242637    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs" pod "b84214b8-3b0f-4902-82f2-1cb36c3f2e99" (UID: "b84214b8-3b0f-4902-82f2-1cb36c3f2e99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:46.341747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341662    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.341747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341697    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htbl\" (UniqueName: \"kubernetes.io/projected/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-kube-api-access-6htbl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.341747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341713    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8xg\" (UniqueName: \"kubernetes.io/projected/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-kube-api-access-dc8xg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.341747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341726    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.341747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341739    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.341747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341750    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341762    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12ab22d-2ee9-4c17-a3da-7001632db2fa-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341772    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341784    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341795    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341807    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341819    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nsw\" (UniqueName: \"kubernetes.io/projected/d12ab22d-2ee9-4c17-a3da-7001632db2fa-kube-api-access-w7nsw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341832    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341844    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341855    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095a98a9-dc08-42d0-8ef6-5bdb796c2c54-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341866    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341877    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12ab22d-2ee9-4c17-a3da-7001632db2fa-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.342159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.341890    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84214b8-3b0f-4902-82f2-1cb36c3f2e99-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:42:46.514675 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.514635    5060 generic.go:356] "Generic (PLEG): container finished" podID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerID="1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084" exitCode=0
Oct 30 12:42:46.514893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.514687    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"
Oct 30 12:42:46.514893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.514722    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" event={"ID":"095a98a9-dc08-42d0-8ef6-5bdb796c2c54","Type":"ContainerDied","Data":"1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084"}
Oct 30 12:42:46.514893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.514762    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh" event={"ID":"095a98a9-dc08-42d0-8ef6-5bdb796c2c54","Type":"ContainerDied","Data":"ba4c32f5ef45014e0013435677274fd3c9d466c49a025eea042605527b5021e6"}
Oct 30 12:42:46.514893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.514779    5060 scope.go:122] "RemoveContainer" containerID="1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084"
Oct 30 12:42:46.517367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.517323    5060 generic.go:356] "Generic (PLEG): container finished" podID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerID="9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463" exitCode=0
Oct 30 12:42:46.517620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.517596    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" event={"ID":"d12ab22d-2ee9-4c17-a3da-7001632db2fa","Type":"ContainerDied","Data":"9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463"}
Oct 30 12:42:46.517726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.517642    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" event={"ID":"d12ab22d-2ee9-4c17-a3da-7001632db2fa","Type":"ContainerDied","Data":"833925d5fe1654c3df3282b4cb41c4e920770aaecfd3e367d9227a5f6a92a815"}
Oct 30 12:42:46.517726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.517717    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"
Oct 30 12:42:46.521065 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.521027    5060 generic.go:356] "Generic (PLEG): container finished" podID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerID="0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526" exitCode=0
Oct 30 12:42:46.521202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.521095    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"
Oct 30 12:42:46.521202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.521177    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" event={"ID":"b84214b8-3b0f-4902-82f2-1cb36c3f2e99","Type":"ContainerDied","Data":"0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526"}
Oct 30 12:42:46.521338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.521205    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5" event={"ID":"b84214b8-3b0f-4902-82f2-1cb36c3f2e99","Type":"ContainerDied","Data":"6b23501218c1a55ed8ce88f0968dd47f693be53cae907bf8aec3dfc60267bb39"}
Oct 30 12:42:46.527861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.527843    5060 scope.go:122] "RemoveContainer" containerID="1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084"
Oct 30 12:42:46.528144 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:46.528121    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084\": container with ID starting with 1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084 not found: ID does not exist" containerID="1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084"
Oct 30 12:42:46.528212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.528166    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084"} err="failed to get container status \"1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084\": rpc error: code = NotFound desc = could not find container \"1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084\": container with ID starting with 1aed8c9eaca4f9384d667419adf57c5d7ca9bd3e927cf1783c663b06bf531084 not found: ID does not exist"
Oct 30 12:42:46.528212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.528187    5060 scope.go:122] "RemoveContainer" containerID="9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463"
Oct 30 12:42:46.538122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.538102    5060 scope.go:122] "RemoveContainer" containerID="9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463"
Oct 30 12:42:46.538384 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:46.538363    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463\": container with ID starting with 9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463 not found: ID does not exist" containerID="9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463"
Oct 30 12:42:46.538479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.538389    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463"} err="failed to get container status \"9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463\": rpc error: code = NotFound desc = could not find container \"9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463\": container with ID starting with 9463118b4ceeb14567278671dfd0d7bdd16f8a810681d1c27c35c56fe4300463 not found: ID does not exist"
Oct 30 12:42:46.538479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.538406    5060 scope.go:122] "RemoveContainer" containerID="0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526"
Oct 30 12:42:46.548208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.548191    5060 scope.go:122] "RemoveContainer" containerID="0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526"
Oct 30 12:42:46.548524 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:46.548505    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526\": container with ID starting with 0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526 not found: ID does not exist" containerID="0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526"
Oct 30 12:42:46.548609 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.548532    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526"} err="failed to get container status \"0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526\": rpc error: code = NotFound desc = could not find container \"0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526\": container with ID starting with 0abac37f69cbb0d65d26e1aeefb9d500b61b1ef6a153d3b7f6d20c756729f526 not found: ID does not exist"
Oct 30 12:42:46.570608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.570583    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"]
Oct 30 12:42:46.574942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.574922    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-849669/simulator-w4-849669-kserve-55b7f65f69-trkn5"]
Oct 30 12:42:46.609969 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.609891    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"]
Oct 30 12:42:46.613601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.613578    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-2-w4-849669-kserve-55ff955464-nxsgh"]
Oct 30 12:42:46.646869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.646839    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"]
Oct 30 12:42:46.654174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:46.654146    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2"]
Oct 30 12:42:47.083069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:47.083009    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/premium-w4-849669-kserve-7799bff564-lx9p2" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:42:47.758943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:47.758904    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:48.335472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:48.335433    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" path="/var/lib/kubelet/pods/095a98a9-dc08-42d0-8ef6-5bdb796c2c54/volumes"
Oct 30 12:42:48.335949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:48.335934    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" path="/var/lib/kubelet/pods/b84214b8-3b0f-4902-82f2-1cb36c3f2e99/volumes"
Oct 30 12:42:48.336358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:48.336345    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" path="/var/lib/kubelet/pods/d12ab22d-2ee9-4c17-a3da-7001632db2fa/volumes"
Oct 30 12:42:49.555880 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:49.555843    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:49.858977 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:42:49.858892    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:42:51.299719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:51.299635    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:51.678513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:51.678378    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:51.817860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:51.817824    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:53.054816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:53.054779    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:54.759431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:54.759395    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:56.214511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.214469    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"]
Oct 30 12:42:56.215102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215086    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main"
Oct 30 12:42:56.215153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215104    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main"
Oct 30 12:42:56.215153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215115    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy"
Oct 30 12:42:56.215153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215120    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy"
Oct 30 12:42:56.215153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215135    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy"
Oct 30 12:42:56.215153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215140    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215160    5060 cpu_manager.go:414] "containerMap: removing container" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215167    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215196    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215200    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215210    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2149452-c5c6-47ff-8700-d304d4a39026" containerName="revoke-keys"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215214    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2149452-c5c6-47ff-8700-d304d4a39026" containerName="revoke-keys"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215301    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215312    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="095a98a9-dc08-42d0-8ef6-5bdb796c2c54" containerName="main"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215327    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12ab22d-2ee9-4c17-a3da-7001632db2fa" containerName="main"
Oct 30 12:42:56.215337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215335    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2149452-c5c6-47ff-8700-d304d4a39026" containerName="revoke-keys"
Oct 30 12:42:56.215699 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.215342    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84214b8-3b0f-4902-82f2-1cb36c3f2e99" containerName="main"
Oct 30 12:42:56.220541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.220518    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.237666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.237616    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"]
Oct 30 12:42:56.242027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.241998    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"]
Oct 30 12:42:56.243077 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.243054    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a37c5b7-b5a1-4389-baa2-692562ae1b96" containerName="istio-proxy"
Oct 30 12:42:56.247340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.244265    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:42:56.247340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.244362    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:42:56.247340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.247104    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.247661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.247639    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:42:56.247739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.247696    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:42:56.266133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.266107    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"]
Oct 30 12:42:56.368435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.368399    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.368435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.368438    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.368733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.368533    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzshv\" (UniqueName: \"kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.368733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.368591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.368733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.368726    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4mr\" (UniqueName: \"kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.368887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.368799    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.469891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.469798    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.469891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.469843    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.469891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.469866    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzshv\" (UniqueName: \"kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.470210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.469899    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.470210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.469994    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4mr\" (UniqueName: \"kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.470210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.470056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.484052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.484023    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:42:56.484421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.484385    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:42:56.491680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.491652    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.491813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.491675    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.493209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.493181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.493429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.493408    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.506355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.506324    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:42:56.514891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.514867    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzshv\" (UniqueName: \"kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv\") pod \"maas-api-revoke-sub-f55322a1-f7n7f\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.515105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.515082    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4mr\" (UniqueName: \"kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr\") pod \"maas-api-revoke-sub-f8c001c8-fd7gt\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.553988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.553963    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:42:56.559568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.559548    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:56.571204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.571181    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:42:56.654299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.654111    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:56.876356 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.876284    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"]
Oct 30 12:42:56.896470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.896427    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"]
Oct 30 12:42:56.900063 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:42:56.900028    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c276f3f_baf0_4bc2_b548_0d48554882c9.slice/crio-5032ac95456aa13a18ea01828731395ae1c202bfe21f48bf11c508266cc056ce WatchSource:0}: Error finding container 5032ac95456aa13a18ea01828731395ae1c202bfe21f48bf11c508266cc056ce: Status 404 returned error can't find the container with id 5032ac95456aa13a18ea01828731395ae1c202bfe21f48bf11c508266cc056ce
Oct 30 12:42:56.941108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:56.941082    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:57.262650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.262620    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:57.621968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.621923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f" event={"ID":"8c276f3f-baf0-4bc2-b548-0d48554882c9","Type":"ContainerStarted","Data":"abb7422ec824231091b19b425fa364dccb37612c4114d26ccbd89d530318d314"}
Oct 30 12:42:57.622183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.621996    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f" event={"ID":"8c276f3f-baf0-4bc2-b548-0d48554882c9","Type":"ContainerStarted","Data":"5032ac95456aa13a18ea01828731395ae1c202bfe21f48bf11c508266cc056ce"}
Oct 30 12:42:57.623965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.623932    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt" event={"ID":"e021ad8a-13c9-4f2a-8282-3a325e84c46e","Type":"ContainerStarted","Data":"9c7b0de26f81ad2083b830d42e6bef488fb57064f7341cec971fe1c6db4fc07f"}
Oct 30 12:42:57.624123 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.623982    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt" event={"ID":"e021ad8a-13c9-4f2a-8282-3a325e84c46e","Type":"ContainerStarted","Data":"641efe12c2d1e7dcbf43364a6480d3c9452dd627da4a21cc25c32bb2c1ed5148"}
Oct 30 12:42:57.665495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.665431    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f" podStartSLOduration=1.665417457 podStartE2EDuration="1.665417457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:57.66279528 +0000 UTC m=+4381.457679775" watchObservedRunningTime="2026-10-30 12:42:57.665417457 +0000 UTC m=+4381.460301996"
Oct 30 12:42:57.706294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:57.706245    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt" podStartSLOduration=1.706232 podStartE2EDuration="1.706232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:57.703154596 +0000 UTC m=+4381.498039089" watchObservedRunningTime="2026-10-30 12:42:57.706232 +0000 UTC m=+4381.501116514"
Oct 30 12:42:58.443891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:58.443851    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:58.634083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:58.634044    5060 generic.go:356] "Generic (PLEG): container finished" podID="8c276f3f-baf0-4bc2-b548-0d48554882c9" containerID="abb7422ec824231091b19b425fa364dccb37612c4114d26ccbd89d530318d314" exitCode=0
Oct 30 12:42:58.634285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:58.634119    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f" event={"ID":"8c276f3f-baf0-4bc2-b548-0d48554882c9","Type":"ContainerDied","Data":"abb7422ec824231091b19b425fa364dccb37612c4114d26ccbd89d530318d314"}
Oct 30 12:42:58.635798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:58.635765    5060 generic.go:356] "Generic (PLEG): container finished" podID="e021ad8a-13c9-4f2a-8282-3a325e84c46e" containerID="9c7b0de26f81ad2083b830d42e6bef488fb57064f7341cec971fe1c6db4fc07f" exitCode=0
Oct 30 12:42:58.635994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:58.635847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt" event={"ID":"e021ad8a-13c9-4f2a-8282-3a325e84c46e","Type":"ContainerDied","Data":"9c7b0de26f81ad2083b830d42e6bef488fb57064f7341cec971fe1c6db4fc07f"}
Oct 30 12:42:59.380054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:59.380012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:59.848155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:59.848134    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:42:59.851571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:42:59.851551    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:43:00.018338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr\" (UniqueName: \"kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr\") pod \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") "
Oct 30 12:43:00.018591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018358    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca\") pod \"8c276f3f-baf0-4bc2-b548-0d48554882c9\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") "
Oct 30 12:43:00.018591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018392    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca\") pod \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") "
Oct 30 12:43:00.018591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018411    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca\") pod \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\" (UID: \"e021ad8a-13c9-4f2a-8282-3a325e84c46e\") "
Oct 30 12:43:00.018591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018513    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv\" (UniqueName: \"kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv\") pod \"8c276f3f-baf0-4bc2-b548-0d48554882c9\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") "
Oct 30 12:43:00.018591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018558    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca\") pod \"8c276f3f-baf0-4bc2-b548-0d48554882c9\" (UID: \"8c276f3f-baf0-4bc2-b548-0d48554882c9\") "
Oct 30 12:43:00.018906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.018825    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca" pod "e021ad8a-13c9-4f2a-8282-3a325e84c46e" (UID: "e021ad8a-13c9-4f2a-8282-3a325e84c46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:00.019177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.019144    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca" pod "8c276f3f-baf0-4bc2-b548-0d48554882c9" (UID: "8c276f3f-baf0-4bc2-b548-0d48554882c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:00.019272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.019241    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:00.021154 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.021120    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca" pod "8c276f3f-baf0-4bc2-b548-0d48554882c9" (UID: "8c276f3f-baf0-4bc2-b548-0d48554882c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:00.021424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.021405    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv" pod "8c276f3f-baf0-4bc2-b548-0d48554882c9" (UID: "8c276f3f-baf0-4bc2-b548-0d48554882c9"). InnerVolumeSpecName "kube-api-access-vzshv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:00.021546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.021441    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca" pod "e021ad8a-13c9-4f2a-8282-3a325e84c46e" (UID: "e021ad8a-13c9-4f2a-8282-3a325e84c46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:00.021546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.021495    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr" pod "e021ad8a-13c9-4f2a-8282-3a325e84c46e" (UID: "e021ad8a-13c9-4f2a-8282-3a325e84c46e"). InnerVolumeSpecName "kube-api-access-qp4mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:00.120985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.120951    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp4mr\" (UniqueName: \"kubernetes.io/projected/e021ad8a-13c9-4f2a-8282-3a325e84c46e-kube-api-access-qp4mr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:00.120985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.120986    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:00.121216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.120996    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e021ad8a-13c9-4f2a-8282-3a325e84c46e-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:00.121216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.121005    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzshv\" (UniqueName: \"kubernetes.io/projected/8c276f3f-baf0-4bc2-b548-0d48554882c9-kube-api-access-vzshv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:00.121216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.121013    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c276f3f-baf0-4bc2-b548-0d48554882c9-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:00.195658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.195435    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:00.343387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.343300    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:00.652964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.652885    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"
Oct 30 12:43:00.653140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.652887    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f" event={"ID":"8c276f3f-baf0-4bc2-b548-0d48554882c9","Type":"ContainerDied","Data":"5032ac95456aa13a18ea01828731395ae1c202bfe21f48bf11c508266cc056ce"}
Oct 30 12:43:00.653140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.653005    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5032ac95456aa13a18ea01828731395ae1c202bfe21f48bf11c508266cc056ce"
Oct 30 12:43:00.654360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.654339    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"
Oct 30 12:43:00.654531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.654379    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt" event={"ID":"e021ad8a-13c9-4f2a-8282-3a325e84c46e","Type":"ContainerDied","Data":"641efe12c2d1e7dcbf43364a6480d3c9452dd627da4a21cc25c32bb2c1ed5148"}
Oct 30 12:43:00.654531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:00.654436    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641efe12c2d1e7dcbf43364a6480d3c9452dd627da4a21cc25c32bb2c1ed5148"
Oct 30 12:43:01.915833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:01.915783    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:03.736858 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:03.736820    5060 scope.go:122] "RemoveContainer" containerID="aeb3ef084162642a7d7885d58a05c169ca2bc4b1bcd84072f2a48cb47a645775"
Oct 30 12:43:03.790179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:03.790152    5060 scope.go:122] "RemoveContainer" containerID="5249e036ec8c06c85764363055ff70ea77fec3db2a0d705b12105a24af3789d7"
Oct 30 12:43:03.790704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:03.790680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:05.554446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:05.554038    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:07.321891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:07.321851    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:08.730221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:08.730182    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:08.886704 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:08.886657    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:09.273033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.272942    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:09.472222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.472189    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:09.496891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.496858    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"]
Oct 30 12:43:09.497772 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.497751    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8c276f3f-baf0-4bc2-b548-0d48554882c9" containerName="revoke-keys"
Oct 30 12:43:09.497869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.497774    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c276f3f-baf0-4bc2-b548-0d48554882c9" containerName="revoke-keys"
Oct 30 12:43:09.497869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.497792    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e021ad8a-13c9-4f2a-8282-3a325e84c46e" containerName="revoke-keys"
Oct 30 12:43:09.497869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.497800    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e021ad8a-13c9-4f2a-8282-3a325e84c46e" containerName="revoke-keys"
Oct 30 12:43:09.498053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.497960    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c276f3f-baf0-4bc2-b548-0d48554882c9" containerName="revoke-keys"
Oct 30 12:43:09.498053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.497973    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e021ad8a-13c9-4f2a-8282-3a325e84c46e" containerName="revoke-keys"
Oct 30 12:43:09.502103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.502079    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.518066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.518040    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:09.520564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.520540    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"]
Oct 30 12:43:09.526767 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.526712    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:09.526896 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.526800    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:09.527519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.527430    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:09.573090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.573057    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcljw\" (UniqueName: \"kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.573244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.573102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.573244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.573220    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.674373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.674343    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.674576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.674443    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcljw\" (UniqueName: \"kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.674576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.674512    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.675124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.675095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.677010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.676985    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.698328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.698296    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcljw\" (UniqueName: \"kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw\") pod \"maas-api-revoke-sub-ce40f8a2-tbd45\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:09.823061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:09.822981    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:10.140064 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:10.140002    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf27c36f_c07a_43b9_bafe_d9b09d062792.slice/crio-cee3e49ceaf3d081cd90d97d07e6e5618d623a1b403d468423ea7439b09b3e2c WatchSource:0}: Error finding container cee3e49ceaf3d081cd90d97d07e6e5618d623a1b403d468423ea7439b09b3e2c: Status 404 returned error can't find the container with id cee3e49ceaf3d081cd90d97d07e6e5618d623a1b403d468423ea7439b09b3e2c
Oct 30 12:43:10.141332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:10.141304    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"]
Oct 30 12:43:10.778500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:10.778436    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45" event={"ID":"af27c36f-c07a-43b9-bafe-d9b09d062792","Type":"ContainerStarted","Data":"832bcd2ea76f6fb6301cd4dbe0c93d7077980a0e8a9521fafb45a2a5dcedec77"}
Oct 30 12:43:10.778715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:10.778557    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45" event={"ID":"af27c36f-c07a-43b9-bafe-d9b09d062792","Type":"ContainerStarted","Data":"cee3e49ceaf3d081cd90d97d07e6e5618d623a1b403d468423ea7439b09b3e2c"}
Oct 30 12:43:10.825222 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:10.825168    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45" podStartSLOduration=1.8251543510000001 podStartE2EDuration="1.825154351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:10.82160794 +0000 UTC m=+4394.616492436" watchObservedRunningTime="2026-10-30 12:43:10.825154351 +0000 UTC m=+4394.620038844"
Oct 30 12:43:11.066243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:11.066143    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:11.790825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:11.790765    5060 generic.go:356] "Generic (PLEG): container finished" podID="af27c36f-c07a-43b9-bafe-d9b09d062792" containerID="832bcd2ea76f6fb6301cd4dbe0c93d7077980a0e8a9521fafb45a2a5dcedec77" exitCode=0
Oct 30 12:43:11.791108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:11.791065    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45" event={"ID":"af27c36f-c07a-43b9-bafe-d9b09d062792","Type":"ContainerDied","Data":"832bcd2ea76f6fb6301cd4dbe0c93d7077980a0e8a9521fafb45a2a5dcedec77"}
Oct 30 12:43:12.965837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:12.965812    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:13.016859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.016806    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:13.018749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.018722    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca\") pod \"af27c36f-c07a-43b9-bafe-d9b09d062792\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") "
Oct 30 12:43:13.018870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.018802    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw\" (UniqueName: \"kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw\") pod \"af27c36f-c07a-43b9-bafe-d9b09d062792\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") "
Oct 30 12:43:13.018986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.018972    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca\") pod \"af27c36f-c07a-43b9-bafe-d9b09d062792\" (UID: \"af27c36f-c07a-43b9-bafe-d9b09d062792\") "
Oct 30 12:43:13.019642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.019610    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca" pod "af27c36f-c07a-43b9-bafe-d9b09d062792" (UID: "af27c36f-c07a-43b9-bafe-d9b09d062792"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:13.021424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.021402    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca" pod "af27c36f-c07a-43b9-bafe-d9b09d062792" (UID: "af27c36f-c07a-43b9-bafe-d9b09d062792"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:13.021760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.021733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw" pod "af27c36f-c07a-43b9-bafe-d9b09d062792" (UID: "af27c36f-c07a-43b9-bafe-d9b09d062792"). InnerVolumeSpecName "kube-api-access-qcljw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:13.121064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.120981    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:13.121064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.121012    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af27c36f-c07a-43b9-bafe-d9b09d062792-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:13.121064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.121021    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcljw\" (UniqueName: \"kubernetes.io/projected/af27c36f-c07a-43b9-bafe-d9b09d062792-kube-api-access-qcljw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:13.811173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.811144    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45"
Oct 30 12:43:13.811396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.811164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce40f8a2-tbd45" event={"ID":"af27c36f-c07a-43b9-bafe-d9b09d062792","Type":"ContainerDied","Data":"cee3e49ceaf3d081cd90d97d07e6e5618d623a1b403d468423ea7439b09b3e2c"}
Oct 30 12:43:13.811396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:13.811230    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee3e49ceaf3d081cd90d97d07e6e5618d623a1b403d468423ea7439b09b3e2c"
Oct 30 12:43:14.411348 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.411306    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"]
Oct 30 12:43:14.412148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.412125    5060 cpu_manager.go:414] "containerMap: removing container" podUID="af27c36f-c07a-43b9-bafe-d9b09d062792" containerName="revoke-keys"
Oct 30 12:43:14.412274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.412150    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af27c36f-c07a-43b9-bafe-d9b09d062792" containerName="revoke-keys"
Oct 30 12:43:14.412343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.412292    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="af27c36f-c07a-43b9-bafe-d9b09d062792" containerName="revoke-keys"
Oct 30 12:43:14.416005 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.415983    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.433771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.433747    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"]
Oct 30 12:43:14.440559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.440534    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:14.440718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.440631    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:14.443195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.443176    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:14.443316 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.443279    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:14.538707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.538675    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.538898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.538720    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx84r\" (UniqueName: \"kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.538898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.538793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.640058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.640021    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.640259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.640076    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx84r\" (UniqueName: \"kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.640259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.640131    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.640741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.640711    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.642617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.642593    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.664074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.664018    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx84r\" (UniqueName: \"kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r\") pod \"maas-api-revoke-sub-18c1f920-2cq49\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:14.714275 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.714230    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:14.735150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:14.735121    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:15.056636 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:15.056589    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b21c73f_3c6f_498f_a0d5_2dc8ff383d7e.slice/crio-7dd19feb3073d42b3eabe5e363d8329702882676bbb526130f9c860a8ccbcce9 WatchSource:0}: Error finding container 7dd19feb3073d42b3eabe5e363d8329702882676bbb526130f9c860a8ccbcce9: Status 404 returned error can't find the container with id 7dd19feb3073d42b3eabe5e363d8329702882676bbb526130f9c860a8ccbcce9
Oct 30 12:43:15.058437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:15.058416    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"]
Oct 30 12:43:15.674997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:15.674918    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:15.834716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:15.834682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49" event={"ID":"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e","Type":"ContainerStarted","Data":"762bbca2d79c96d35684ddb3b1714731eb5480e94b46976af61f0360e3b93fad"}
Oct 30 12:43:15.834883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:15.834736    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49" event={"ID":"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e","Type":"ContainerStarted","Data":"7dd19feb3073d42b3eabe5e363d8329702882676bbb526130f9c860a8ccbcce9"}
Oct 30 12:43:15.854347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:15.854311    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:15.951591 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:15.951543    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49" podStartSLOduration=1.951529185 podStartE2EDuration="1.951529185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:15.946688977 +0000 UTC m=+4399.741573472" watchObservedRunningTime="2026-10-30 12:43:15.951529185 +0000 UTC m=+4399.746413680"
Oct 30 12:43:16.548870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:16.548835    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:16.843818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:16.843724    5060 generic.go:356] "Generic (PLEG): container finished" podID="4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" containerID="762bbca2d79c96d35684ddb3b1714731eb5480e94b46976af61f0360e3b93fad" exitCode=0
Oct 30 12:43:16.844250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:16.843819    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49" event={"ID":"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e","Type":"ContainerDied","Data":"762bbca2d79c96d35684ddb3b1714731eb5480e94b46976af61f0360e3b93fad"}
Oct 30 12:43:18.026161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.026128    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:18.182680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.182577    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca\") pod \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") "
Oct 30 12:43:18.182884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.182697    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r\" (UniqueName: \"kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r\") pod \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") "
Oct 30 12:43:18.182884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.182829    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca\") pod \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\" (UID: \"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e\") "
Oct 30 12:43:18.183040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.183013    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca" pod "4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" (UID: "4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:18.183393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.183365    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:18.185816 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.185786    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca" pod "4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" (UID: "4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:18.185972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.185815    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r" pod "4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" (UID: "4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e"). InnerVolumeSpecName "kube-api-access-qx84r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:18.285429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.285388    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx84r\" (UniqueName: \"kubernetes.io/projected/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-kube-api-access-qx84r\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:18.285429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.285433    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:18.319135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.319085    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:18.860432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.860401    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49"
Oct 30 12:43:18.860668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.860427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18c1f920-2cq49" event={"ID":"4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e","Type":"ContainerDied","Data":"7dd19feb3073d42b3eabe5e363d8329702882676bbb526130f9c860a8ccbcce9"}
Oct 30 12:43:18.860668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:18.860511    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd19feb3073d42b3eabe5e363d8329702882676bbb526130f9c860a8ccbcce9"
Oct 30 12:43:19.863783 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:43:19.863743    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:43:19.975477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:19.968758    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"]
Oct 30 12:43:19.975477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:19.969741    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" containerName="revoke-keys"
Oct 30 12:43:19.975477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:19.969762    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" containerName="revoke-keys"
Oct 30 12:43:19.975477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:19.969954    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e" containerName="revoke-keys"
Oct 30 12:43:19.990475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:19.981938    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.002552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:19.997398    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"]
Oct 30 12:43:20.002552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.002380    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:20.002552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.002488    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:20.019862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.014398    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:20.019862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.014501    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:20.083370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.083330    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:20.112590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.112541    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.112784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.112628    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.112838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.112787    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvcx\" (UniqueName: \"kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.214654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.214614    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.214844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.214686    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.214844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.214780    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvcx\" (UniqueName: \"kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.215296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.215274    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.217248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.217225    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.238792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.238766    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvcx\" (UniqueName: \"kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx\") pod \"maas-api-revoke-sub-a2d7d9c6-45nq8\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.323422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.323388    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:20.634117 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:20.634079    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061ffc44_8b4b_4598_b062_a95fe670f510.slice/crio-ea9df02b8496966d1418e0d84ec198a46f3ab97643fbc18c67a5450c10f6b93c WatchSource:0}: Error finding container ea9df02b8496966d1418e0d84ec198a46f3ab97643fbc18c67a5450c10f6b93c: Status 404 returned error can't find the container with id ea9df02b8496966d1418e0d84ec198a46f3ab97643fbc18c67a5450c10f6b93c
Oct 30 12:43:20.635290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.635252    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"]
Oct 30 12:43:20.877232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:20.877193    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8" event={"ID":"061ffc44-8b4b-4598-b062-a95fe670f510","Type":"ContainerStarted","Data":"ea9df02b8496966d1418e0d84ec198a46f3ab97643fbc18c67a5450c10f6b93c"}
Oct 30 12:43:21.874011 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:21.873926    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:21.889747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:21.889706    5060 generic.go:356] "Generic (PLEG): container finished" podID="061ffc44-8b4b-4598-b062-a95fe670f510" containerID="404696714dafe4d892a08b5df8f702543d4f13ca7e1eeb2002316839ad17f3dc" exitCode=0
Oct 30 12:43:21.890241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:21.889881    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8" event={"ID":"061ffc44-8b4b-4598-b062-a95fe670f510","Type":"ContainerDied","Data":"404696714dafe4d892a08b5df8f702543d4f13ca7e1eeb2002316839ad17f3dc"}
Oct 30 12:43:23.072654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.072628    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:23.150745 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.150633    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx\" (UniqueName: \"kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx\") pod \"061ffc44-8b4b-4598-b062-a95fe670f510\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") "
Oct 30 12:43:23.150986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.150751    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca\") pod \"061ffc44-8b4b-4598-b062-a95fe670f510\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") "
Oct 30 12:43:23.150986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.150875    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca\") pod \"061ffc44-8b4b-4598-b062-a95fe670f510\" (UID: \"061ffc44-8b4b-4598-b062-a95fe670f510\") "
Oct 30 12:43:23.151587 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.151557    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca" pod "061ffc44-8b4b-4598-b062-a95fe670f510" (UID: "061ffc44-8b4b-4598-b062-a95fe670f510"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:23.153313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.153289    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca" pod "061ffc44-8b4b-4598-b062-a95fe670f510" (UID: "061ffc44-8b4b-4598-b062-a95fe670f510"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:23.154174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.154152    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx" pod "061ffc44-8b4b-4598-b062-a95fe670f510" (UID: "061ffc44-8b4b-4598-b062-a95fe670f510"). InnerVolumeSpecName "kube-api-access-6fvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:23.252323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.252287    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fvcx\" (UniqueName: \"kubernetes.io/projected/061ffc44-8b4b-4598-b062-a95fe670f510-kube-api-access-6fvcx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:23.252323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.252318    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:23.252545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.252332    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061ffc44-8b4b-4598-b062-a95fe670f510-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:23.609893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.609848    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:23.911523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.911224    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8"
Oct 30 12:43:23.911523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.911228    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d7d9c6-45nq8" event={"ID":"061ffc44-8b4b-4598-b062-a95fe670f510","Type":"ContainerDied","Data":"ea9df02b8496966d1418e0d84ec198a46f3ab97643fbc18c67a5450c10f6b93c"}
Oct 30 12:43:23.911523 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:23.911367    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9df02b8496966d1418e0d84ec198a46f3ab97643fbc18c67a5450c10f6b93c"
Oct 30 12:43:24.836388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.836356    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"]
Oct 30 12:43:24.836970 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.836954    5060 cpu_manager.go:414] "containerMap: removing container" podUID="061ffc44-8b4b-4598-b062-a95fe670f510" containerName="revoke-keys"
Oct 30 12:43:24.837022 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.836972    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061ffc44-8b4b-4598-b062-a95fe670f510" containerName="revoke-keys"
Oct 30 12:43:24.837090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.837081    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="061ffc44-8b4b-4598-b062-a95fe670f510" containerName="revoke-keys"
Oct 30 12:43:24.840758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.840737    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:24.862599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.862564    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"]
Oct 30 12:43:24.866017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.865979    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:24.866203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.866069    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:24.866307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.866201    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:24.868034 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.868014    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:24.974986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.974952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:24.975170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.975000    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzpdz\" (UniqueName: \"kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:24.975170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:24.975155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.076592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.076547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.076592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.076606    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzpdz\" (UniqueName: \"kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.076881 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.076710    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.077373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.077349    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.079087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.079061    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.100102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.100037    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzpdz\" (UniqueName: \"kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz\") pod \"maas-api-revoke-sub-a304d978-jx7p2\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.156119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.156092    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:25.285091 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.285050    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:25.463821 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:25.463793    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949e431c_70d8_4c9d_aa44_2def91b2ef2f.slice/crio-b6c2e93ef871cd8590428ffe1874debabcdcffb7c2305dc84c0e272d8daece06 WatchSource:0}: Error finding container b6c2e93ef871cd8590428ffe1874debabcdcffb7c2305dc84c0e272d8daece06: Status 404 returned error can't find the container with id b6c2e93ef871cd8590428ffe1874debabcdcffb7c2305dc84c0e272d8daece06
Oct 30 12:43:25.466052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.466031    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"]
Oct 30 12:43:25.702617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.702579    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"]
Oct 30 12:43:25.708288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.708263    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.737674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.737638    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f1b14f-openshift-default-dockercfg-gpx6g\""
Oct 30 12:43:25.742120 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.742094    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"]
Oct 30 12:43:25.791333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791302    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791473    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791541    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp8bx\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791633    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791709 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791690    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791738    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791861    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.791952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791932    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.792012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.791977    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893324    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893366    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893417    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893486    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893599    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp8bx\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893702    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893747    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.893809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893785    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.894228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893828    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.894228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.893852    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.894345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.894221    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.894345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.894331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.894494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.894381    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.894657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.894615    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.896015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.895991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.896346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.896328    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.916918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.916895    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.917069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.916979    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp8bx\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:25.932113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.932081    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2" event={"ID":"949e431c-70d8-4c9d-aa44-2def91b2ef2f","Type":"ContainerStarted","Data":"8840afd1a0e6714c990a9506ef85283bbc54adc0ca241e3f2e8fc175a4a3c653"}
Oct 30 12:43:25.932232 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.932133    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2" event={"ID":"949e431c-70d8-4c9d-aa44-2def91b2ef2f","Type":"ContainerStarted","Data":"b6c2e93ef871cd8590428ffe1874debabcdcffb7c2305dc84c0e272d8daece06"}
Oct 30 12:43:25.976185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:25.976075    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2" podStartSLOduration=1.976060444 podStartE2EDuration="1.976060444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:25.974992195 +0000 UTC m=+4409.769876716" watchObservedRunningTime="2026-10-30 12:43:25.976060444 +0000 UTC m=+4409.770944938"
Oct 30 12:43:26.024092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.024051    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:26.362270 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:26.362227    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf0fbc9_54b9_46d5_a4fe_04ed4943dbde.slice/crio-9153c9d54f622f4162ef21deebfcab81d5dd8f82831960ee97232c130a2b7e1c WatchSource:0}: Error finding container 9153c9d54f622f4162ef21deebfcab81d5dd8f82831960ee97232c130a2b7e1c: Status 404 returned error can't find the container with id 9153c9d54f622f4162ef21deebfcab81d5dd8f82831960ee97232c130a2b7e1c
Oct 30 12:43:26.363390 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.363362    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"]
Oct 30 12:43:26.365468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.365410    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:43:26.365596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.365537    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:43:26.365596 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.365586    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:43:26.942159 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.942121    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" event={"ID":"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde","Type":"ContainerStarted","Data":"69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594"}
Oct 30 12:43:26.942754 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.942184    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" event={"ID":"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde","Type":"ContainerStarted","Data":"9153c9d54f622f4162ef21deebfcab81d5dd8f82831960ee97232c130a2b7e1c"}
Oct 30 12:43:26.943932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.943909    5060 generic.go:356] "Generic (PLEG): container finished" podID="949e431c-70d8-4c9d-aa44-2def91b2ef2f" containerID="8840afd1a0e6714c990a9506ef85283bbc54adc0ca241e3f2e8fc175a4a3c653" exitCode=0
Oct 30 12:43:26.944037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.944004    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2" event={"ID":"949e431c-70d8-4c9d-aa44-2def91b2ef2f","Type":"ContainerDied","Data":"8840afd1a0e6714c990a9506ef85283bbc54adc0ca241e3f2e8fc175a4a3c653"}
Oct 30 12:43:26.977258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:26.977222    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:27.024283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:27.024245    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:27.025790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:27.025764    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 12:43:27.025951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:27.025809    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 12:43:27.045805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:27.045744    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podStartSLOduration=2.045729259 podStartE2EDuration="2.045729259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:27.002170836 +0000 UTC m=+4410.797055331" watchObservedRunningTime="2026-10-30 12:43:27.045729259 +0000 UTC m=+4410.840613756"
Oct 30 12:43:28.025195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.025150    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 12:43:28.025752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.025210    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 12:43:28.167833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.167805    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:28.235139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.235042    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca\") pod \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") "
Oct 30 12:43:28.235315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.235244    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz\" (UniqueName: \"kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz\") pod \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") "
Oct 30 12:43:28.235393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.235351    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca\") pod \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\" (UID: \"949e431c-70d8-4c9d-aa44-2def91b2ef2f\") "
Oct 30 12:43:28.235874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.235843    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca" pod "949e431c-70d8-4c9d-aa44-2def91b2ef2f" (UID: "949e431c-70d8-4c9d-aa44-2def91b2ef2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:28.236075 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.236054    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:28.237579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.237544    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca" pod "949e431c-70d8-4c9d-aa44-2def91b2ef2f" (UID: "949e431c-70d8-4c9d-aa44-2def91b2ef2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:28.237951 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.237891    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz" pod "949e431c-70d8-4c9d-aa44-2def91b2ef2f" (UID: "949e431c-70d8-4c9d-aa44-2def91b2ef2f"). InnerVolumeSpecName "kube-api-access-gzpdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:28.337228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.337190    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949e431c-70d8-4c9d-aa44-2def91b2ef2f-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:28.337228 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.337220    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzpdz\" (UniqueName: \"kubernetes.io/projected/949e431c-70d8-4c9d-aa44-2def91b2ef2f-kube-api-access-gzpdz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:28.754822 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.754786    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:28.961019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.960990    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2"
Oct 30 12:43:28.961180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.961037    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a304d978-jx7p2" event={"ID":"949e431c-70d8-4c9d-aa44-2def91b2ef2f","Type":"ContainerDied","Data":"b6c2e93ef871cd8590428ffe1874debabcdcffb7c2305dc84c0e272d8daece06"}
Oct 30 12:43:28.961180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:28.961088    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6c2e93ef871cd8590428ffe1874debabcdcffb7c2305dc84c0e272d8daece06"
Oct 30 12:43:29.024967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:29.024870    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 12:43:29.024967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:29.024932    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 12:43:30.025226 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:30.025190    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 12:43:30.025703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:30.025260    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 12:43:30.265799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:30.265763    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:30.850382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:30.850343    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:30.957492 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:30.957418    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"]
Oct 30 12:43:30.971567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:30.971536    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-849669-vsdbh"]
Oct 30 12:43:31.017304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:31.017105    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:31.036164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:31.029006    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 12:43:31.036164 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:31.029067    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 12:43:31.302604 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:31.302556    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:31.547429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:31.547375    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:32.024992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:32.024951    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 12:43:32.025179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:32.025013    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 12:43:32.335636 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:32.335560    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2149452-c5c6-47ff-8700-d304d4a39026" path="/var/lib/kubelet/pods/f2149452-c5c6-47ff-8700-d304d4a39026/volumes"
Oct 30 12:43:33.051810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:33.051772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:33.172933 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:33.172901    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:33.173777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:33.173752    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:33.175098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:33.175078    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:43:34.751182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:34.751127    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:35.685974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.685939    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"]
Oct 30 12:43:35.686560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.686541    5060 cpu_manager.go:414] "containerMap: removing container" podUID="949e431c-70d8-4c9d-aa44-2def91b2ef2f" containerName="revoke-keys"
Oct 30 12:43:35.686560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.686558    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949e431c-70d8-4c9d-aa44-2def91b2ef2f" containerName="revoke-keys"
Oct 30 12:43:35.686738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.686725    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="949e431c-70d8-4c9d-aa44-2def91b2ef2f" containerName="revoke-keys"
Oct 30 12:43:35.689733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.689717    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.711040 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.711012    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"]
Oct 30 12:43:35.716946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.716919    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:35.717218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.717183    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:35.717387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.717364    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:35.717539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.717482    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:35.780254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.780210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.780949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.780917    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.781138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.781112    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xpw\" (UniqueName: \"kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.882141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.882103    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xpw\" (UniqueName: \"kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.882330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.882309    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.882399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.882368    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.882997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.882961    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.884832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.884811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:35.906219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:35.906189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xpw\" (UniqueName: \"kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw\") pod \"maas-api-revoke-sub-8ee3a86b-ckdlx\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:36.013110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:36.013078    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:36.324283 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:36.324245    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e86adc1_ff0a_4daa_aa66_fece36e33add.slice/crio-4aa97dd4a8d0c764debd33fa1fe3c37dcc73c332c2dbfee5fb8f51c373f77426 WatchSource:0}: Error finding container 4aa97dd4a8d0c764debd33fa1fe3c37dcc73c332c2dbfee5fb8f51c373f77426: Status 404 returned error can't find the container with id 4aa97dd4a8d0c764debd33fa1fe3c37dcc73c332c2dbfee5fb8f51c373f77426
Oct 30 12:43:36.325054 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:36.325035    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"]
Oct 30 12:43:36.560482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:36.560013    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:37.051318 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:37.051276    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx" event={"ID":"6e86adc1-ff0a-4daa-aa66-fece36e33add","Type":"ContainerStarted","Data":"3c198c8d7c7071b37afcbec7a3d9213b10889b4a4c343a0eb0b00159509dedd5"}
Oct 30 12:43:37.051828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:37.051334    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx" event={"ID":"6e86adc1-ff0a-4daa-aa66-fece36e33add","Type":"ContainerStarted","Data":"4aa97dd4a8d0c764debd33fa1fe3c37dcc73c332c2dbfee5fb8f51c373f77426"}
Oct 30 12:43:37.100125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:37.100057    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx" podStartSLOduration=2.100036885 podStartE2EDuration="2.100036885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:37.096912348 +0000 UTC m=+4420.891796842" watchObservedRunningTime="2026-10-30 12:43:37.100036885 +0000 UTC m=+4420.894921384"
Oct 30 12:43:38.018241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:38.018202    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:38.066663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:38.066630    5060 generic.go:356] "Generic (PLEG): container finished" podID="6e86adc1-ff0a-4daa-aa66-fece36e33add" containerID="3c198c8d7c7071b37afcbec7a3d9213b10889b4a4c343a0eb0b00159509dedd5" exitCode=0
Oct 30 12:43:38.067102 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:38.066675    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx" event={"ID":"6e86adc1-ff0a-4daa-aa66-fece36e33add","Type":"ContainerDied","Data":"3c198c8d7c7071b37afcbec7a3d9213b10889b4a4c343a0eb0b00159509dedd5"}
Oct 30 12:43:38.165347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:38.165310    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:38.366026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:38.365929    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:38.508165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:38.508125    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:39.243411 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.243385    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:39.342325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.342239    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca\") pod \"6e86adc1-ff0a-4daa-aa66-fece36e33add\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") "
Oct 30 12:43:39.342528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.342338    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw\" (UniqueName: \"kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw\") pod \"6e86adc1-ff0a-4daa-aa66-fece36e33add\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") "
Oct 30 12:43:39.342528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.342395    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca\") pod \"6e86adc1-ff0a-4daa-aa66-fece36e33add\" (UID: \"6e86adc1-ff0a-4daa-aa66-fece36e33add\") "
Oct 30 12:43:39.342806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.342778    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca" pod "6e86adc1-ff0a-4daa-aa66-fece36e33add" (UID: "6e86adc1-ff0a-4daa-aa66-fece36e33add"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:39.342946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.342928    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:39.344619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.344589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca" pod "6e86adc1-ff0a-4daa-aa66-fece36e33add" (UID: "6e86adc1-ff0a-4daa-aa66-fece36e33add"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:39.344747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.344656    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw" pod "6e86adc1-ff0a-4daa-aa66-fece36e33add" (UID: "6e86adc1-ff0a-4daa-aa66-fece36e33add"). InnerVolumeSpecName "kube-api-access-t9xpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:39.443954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.443921    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xpw\" (UniqueName: \"kubernetes.io/projected/6e86adc1-ff0a-4daa-aa66-fece36e33add-kube-api-access-t9xpw\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:39.443954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:39.443951    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e86adc1-ff0a-4daa-aa66-fece36e33add-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:40.083932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:40.083901    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx"
Oct 30 12:43:40.084144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:40.083945    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee3a86b-ckdlx" event={"ID":"6e86adc1-ff0a-4daa-aa66-fece36e33add","Type":"ContainerDied","Data":"4aa97dd4a8d0c764debd33fa1fe3c37dcc73c332c2dbfee5fb8f51c373f77426"}
Oct 30 12:43:40.084144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:40.083991    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa97dd4a8d0c764debd33fa1fe3c37dcc73c332c2dbfee5fb8f51c373f77426"
Oct 30 12:43:40.122122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:40.122087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:41.781922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:41.781883    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:43.507117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:43.507076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:45.254325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:45.254285    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:46.184209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.184172    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"]
Oct 30 12:43:46.184849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.184831    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6e86adc1-ff0a-4daa-aa66-fece36e33add" containerName="revoke-keys"
Oct 30 12:43:46.184849 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.184849    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e86adc1-ff0a-4daa-aa66-fece36e33add" containerName="revoke-keys"
Oct 30 12:43:46.185037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.184965    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e86adc1-ff0a-4daa-aa66-fece36e33add" containerName="revoke-keys"
Oct 30 12:43:46.188742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.188720    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.208106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.208072    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"]
Oct 30 12:43:46.216334 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.216303    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:46.216558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.216527    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:46.216558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.216484    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:46.216558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.216405    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:46.328843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.328811    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.329378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.328978    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2r48\" (UniqueName: \"kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.329378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.329044    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.430419 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.430381    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2r48\" (UniqueName: \"kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.430658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.430438    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.430658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.430541    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.431166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.431145    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.433094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.433074    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.454401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.454378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2r48\" (UniqueName: \"kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48\") pod \"maas-api-revoke-sub-28e0340e-nlkxs\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.507418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.507385    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:46.818863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.818778    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"]
Oct 30 12:43:46.825817 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:46.825785    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f8fe96_6db6_455c_89f9_5beba9369c34.slice/crio-d10f2ea56aed6192e4d2afea126e2a2d72afce7abf88151c2c661802f0766f09 WatchSource:0}: Error finding container d10f2ea56aed6192e4d2afea126e2a2d72afce7abf88151c2c661802f0766f09: Status 404 returned error can't find the container with id d10f2ea56aed6192e4d2afea126e2a2d72afce7abf88151c2c661802f0766f09
Oct 30 12:43:46.958229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:46.958195    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:47.144708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:47.144624    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs" event={"ID":"54f8fe96-6db6-455c-89f9-5beba9369c34","Type":"ContainerStarted","Data":"d10f2ea56aed6192e4d2afea126e2a2d72afce7abf88151c2c661802f0766f09"}
Oct 30 12:43:48.158813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.158422    5060 generic.go:356] "Generic (PLEG): container finished" podID="54f8fe96-6db6-455c-89f9-5beba9369c34" containerID="353079a43c6132f8cf25777375551697d9cd03602b2030f34f5aef730fca3bbf" exitCode=0
Oct 30 12:43:48.158813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.158714    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs" event={"ID":"54f8fe96-6db6-455c-89f9-5beba9369c34","Type":"ContainerDied","Data":"353079a43c6132f8cf25777375551697d9cd03602b2030f34f5aef730fca3bbf"}
Oct 30 12:43:48.274384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.274336    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:48.549337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.549301    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"]
Oct 30 12:43:48.554498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.554437    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.572374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.572342    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"]
Oct 30 12:43:48.686137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.686107    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxhm\" (UniqueName: \"kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.686307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.686170    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.686417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.686392    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.785434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.785389    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:48.787776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.787752    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.787908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.787824    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxhm\" (UniqueName: \"kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.787908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.787883    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.788639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.788607    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.790535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.790484    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.827533 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.820836    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxhm\" (UniqueName: \"kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm\") pod \"maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:48.878916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:48.878682    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:49.235950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.235334    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"]
Oct 30 12:43:49.339191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.339157    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:49.509877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.509834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca\") pod \"54f8fe96-6db6-455c-89f9-5beba9369c34\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") "
Oct 30 12:43:49.510118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.509986    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca\") pod \"54f8fe96-6db6-455c-89f9-5beba9369c34\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") "
Oct 30 12:43:49.510118 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.510046    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48\" (UniqueName: \"kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48\") pod \"54f8fe96-6db6-455c-89f9-5beba9369c34\" (UID: \"54f8fe96-6db6-455c-89f9-5beba9369c34\") "
Oct 30 12:43:49.510639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.510605    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca" pod "54f8fe96-6db6-455c-89f9-5beba9369c34" (UID: "54f8fe96-6db6-455c-89f9-5beba9369c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:49.512463 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.512418    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca" pod "54f8fe96-6db6-455c-89f9-5beba9369c34" (UID: "54f8fe96-6db6-455c-89f9-5beba9369c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:49.513267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.513236    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48" pod "54f8fe96-6db6-455c-89f9-5beba9369c34" (UID: "54f8fe96-6db6-455c-89f9-5beba9369c34"). InnerVolumeSpecName "kube-api-access-l2r48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:49.611991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.611952    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:49.611991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.611985    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2r48\" (UniqueName: \"kubernetes.io/projected/54f8fe96-6db6-455c-89f9-5beba9369c34-kube-api-access-l2r48\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:49.611991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:49.611996    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f8fe96-6db6-455c-89f9-5beba9369c34-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:49.858777 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:43:49.858739    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:43:50.181856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.179534    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs"
Oct 30 12:43:50.181856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.179587    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e0340e-nlkxs" event={"ID":"54f8fe96-6db6-455c-89f9-5beba9369c34","Type":"ContainerDied","Data":"d10f2ea56aed6192e4d2afea126e2a2d72afce7abf88151c2c661802f0766f09"}
Oct 30 12:43:50.181856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.179665    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d10f2ea56aed6192e4d2afea126e2a2d72afce7abf88151c2c661802f0766f09"
Oct 30 12:43:50.181856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.181703    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r" event={"ID":"3579ae66-6544-4539-9ede-0ee800706f99","Type":"ContainerStarted","Data":"02d3d61786cbd0e75bb6a81a4c195ef9514c1191a75d186ba37e0b704b5da77a"}
Oct 30 12:43:50.181856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.181743    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r" event={"ID":"3579ae66-6544-4539-9ede-0ee800706f99","Type":"ContainerStarted","Data":"94934e3a409908cbbca5a5370a7058fabb9e5432567d219e28dc44fc39069a47"}
Oct 30 12:43:50.225519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.225437    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r" podStartSLOduration=2.225419772 podStartE2EDuration="2.225419772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:50.222394356 +0000 UTC m=+4434.017278851" watchObservedRunningTime="2026-10-30 12:43:50.225419772 +0000 UTC m=+4434.020304267"
Oct 30 12:43:50.720722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:50.720305    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:51.193382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:51.193285    5060 generic.go:356] "Generic (PLEG): container finished" podID="3579ae66-6544-4539-9ede-0ee800706f99" containerID="02d3d61786cbd0e75bb6a81a4c195ef9514c1191a75d186ba37e0b704b5da77a" exitCode=0
Oct 30 12:43:51.193572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:51.193380    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r" event={"ID":"3579ae66-6544-4539-9ede-0ee800706f99","Type":"ContainerDied","Data":"02d3d61786cbd0e75bb6a81a4c195ef9514c1191a75d186ba37e0b704b5da77a"}
Oct 30 12:43:52.358753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.358719    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:52.390979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.390941    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:52.447080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.447051    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca\") pod \"3579ae66-6544-4539-9ede-0ee800706f99\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") "
Oct 30 12:43:52.447301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.447107    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca\") pod \"3579ae66-6544-4539-9ede-0ee800706f99\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") "
Oct 30 12:43:52.447301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.447165    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm\" (UniqueName: \"kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm\") pod \"3579ae66-6544-4539-9ede-0ee800706f99\" (UID: \"3579ae66-6544-4539-9ede-0ee800706f99\") "
Oct 30 12:43:52.447659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.447627    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca" pod "3579ae66-6544-4539-9ede-0ee800706f99" (UID: "3579ae66-6544-4539-9ede-0ee800706f99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:52.449655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.449626    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm" pod "3579ae66-6544-4539-9ede-0ee800706f99" (UID: "3579ae66-6544-4539-9ede-0ee800706f99"). InnerVolumeSpecName "kube-api-access-qsxhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:52.449893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.449872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca" pod "3579ae66-6544-4539-9ede-0ee800706f99" (UID: "3579ae66-6544-4539-9ede-0ee800706f99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:52.549139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.549098    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:52.549139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.549130    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3579ae66-6544-4539-9ede-0ee800706f99-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:52.549139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:52.549140    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxhm\" (UniqueName: \"kubernetes.io/projected/3579ae66-6544-4539-9ede-0ee800706f99-kube-api-access-qsxhm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:53.214035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:53.214002    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"
Oct 30 12:43:53.214251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:53.214048    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r" event={"ID":"3579ae66-6544-4539-9ede-0ee800706f99","Type":"ContainerDied","Data":"94934e3a409908cbbca5a5370a7058fabb9e5432567d219e28dc44fc39069a47"}
Oct 30 12:43:53.214251 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:53.214143    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94934e3a409908cbbca5a5370a7058fabb9e5432567d219e28dc44fc39069a47"
Oct 30 12:43:54.173272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:54.173228    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:55.704722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.704677    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"]
Oct 30 12:43:55.705635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.705611    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3579ae66-6544-4539-9ede-0ee800706f99" containerName="revoke-keys"
Oct 30 12:43:55.705808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.705637    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3579ae66-6544-4539-9ede-0ee800706f99" containerName="revoke-keys"
Oct 30 12:43:55.705808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.705671    5060 cpu_manager.go:414] "containerMap: removing container" podUID="54f8fe96-6db6-455c-89f9-5beba9369c34" containerName="revoke-keys"
Oct 30 12:43:55.705808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.705680    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f8fe96-6db6-455c-89f9-5beba9369c34" containerName="revoke-keys"
Oct 30 12:43:55.705998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.705850    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3579ae66-6544-4539-9ede-0ee800706f99" containerName="revoke-keys"
Oct 30 12:43:55.705998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.705868    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f8fe96-6db6-455c-89f9-5beba9369c34" containerName="revoke-keys"
Oct 30 12:43:55.710532 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.710507    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.725195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.725165    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"]
Oct 30 12:43:55.731140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.731116    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:43:55.731534 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.731144    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:43:55.731689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.731250    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:43:55.731689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.731310    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:43:55.791033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.791004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.791249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.791073    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.791323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.791256    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj7j\" (UniqueName: \"kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.859579 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.859539    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:55.895427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.895376    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.895718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.895521    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.895718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.895687    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj7j\" (UniqueName: \"kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.896979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.896949    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.911112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.911085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:55.920099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:55.920076    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj7j\" (UniqueName: \"kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j\") pod \"maas-api-revoke-sub-8e35ae98-86wg4\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:56.029848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:56.029752    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:56.336403 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:43:56.336372    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b9e9b9_53d0_4945_bdca_df9bcfd8e38e.slice/crio-29a05f204af2cc858c01c6cf96d0b8da2888240cb1dcf7f9fec4840d01811a67 WatchSource:0}: Error finding container 29a05f204af2cc858c01c6cf96d0b8da2888240cb1dcf7f9fec4840d01811a67: Status 404 returned error can't find the container with id 29a05f204af2cc858c01c6cf96d0b8da2888240cb1dcf7f9fec4840d01811a67
Oct 30 12:43:56.339100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:56.339076    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"]
Oct 30 12:43:57.073124 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:57.073086    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:57.253531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:57.253495    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4" event={"ID":"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e","Type":"ContainerStarted","Data":"5aa4c7a023488e42f1da6f61c12aa81b3fad67ec75b40205ba19a85bd58b7881"}
Oct 30 12:43:57.253700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:57.253549    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4" event={"ID":"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e","Type":"ContainerStarted","Data":"29a05f204af2cc858c01c6cf96d0b8da2888240cb1dcf7f9fec4840d01811a67"}
Oct 30 12:43:57.296244 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:57.296179    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4" podStartSLOduration=2.29616526 podStartE2EDuration="2.29616526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:57.293627175 +0000 UTC m=+4441.088511669" watchObservedRunningTime="2026-10-30 12:43:57.29616526 +0000 UTC m=+4441.091049753"
Oct 30 12:43:57.306203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:57.306169    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:57.655865 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:57.655766    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:58.263214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:58.263178    5060 generic.go:356] "Generic (PLEG): container finished" podID="40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" containerID="5aa4c7a023488e42f1da6f61c12aa81b3fad67ec75b40205ba19a85bd58b7881" exitCode=0
Oct 30 12:43:58.263670 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:58.263260    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4" event={"ID":"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e","Type":"ContainerDied","Data":"5aa4c7a023488e42f1da6f61c12aa81b3fad67ec75b40205ba19a85bd58b7881"}
Oct 30 12:43:59.014888 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.014854    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"]
Oct 30 12:43:59.190426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.190375    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:59.280657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.280554    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"]
Oct 30 12:43:59.336885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.336805    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:59.499121 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.498912    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:59.593693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.593666    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:43:59.646125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.646095    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j\" (UniqueName: \"kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j\") pod \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") "
Oct 30 12:43:59.646333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.646169    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca\") pod \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") "
Oct 30 12:43:59.646412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.646359    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca\") pod \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\" (UID: \"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e\") "
Oct 30 12:43:59.646576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.646549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca" pod "40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" (UID: "40b9e9b9-53d0-4945-bdca-df9bcfd8e38e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:59.646871 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.646828    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:59.649100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.649066    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca" pod "40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" (UID: "40b9e9b9-53d0-4945-bdca-df9bcfd8e38e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:59.649621 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.649595    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j" pod "40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" (UID: "40b9e9b9-53d0-4945-bdca-df9bcfd8e38e"). InnerVolumeSpecName "kube-api-access-pzj7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:59.753476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.753326    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzj7j\" (UniqueName: \"kubernetes.io/projected/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-kube-api-access-pzj7j\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:43:59.753476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:43:59.753366    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b9e9b9-53d0-4945-bdca-df9bcfd8e38e-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:00.301179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:00.301145    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4"
Oct 30 12:44:00.301179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:00.301167    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e35ae98-86wg4" event={"ID":"40b9e9b9-53d0-4945-bdca-df9bcfd8e38e","Type":"ContainerDied","Data":"29a05f204af2cc858c01c6cf96d0b8da2888240cb1dcf7f9fec4840d01811a67"}
Oct 30 12:44:00.301800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:00.301222    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a05f204af2cc858c01c6cf96d0b8da2888240cb1dcf7f9fec4840d01811a67"
Oct 30 12:44:00.903535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:00.903498    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:01.314476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.314162    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:01.356680 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.356647    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"]
Oct 30 12:44:01.484739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.484699    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:01.607623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.607541    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"]
Oct 30 12:44:01.611427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.611405    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-67e819-fgt8r"]
Oct 30 12:44:01.638141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.638107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:01.748979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.748943    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"]
Oct 30 12:44:01.749631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.749615    5060 cpu_manager.go:414] "containerMap: removing container" podUID="40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" containerName="revoke-keys"
Oct 30 12:44:01.749700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.749634    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" containerName="revoke-keys"
Oct 30 12:44:01.749755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.749746    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b9e9b9-53d0-4945-bdca-df9bcfd8e38e" containerName="revoke-keys"
Oct 30 12:44:01.753629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.753607    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.768744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.768716    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:44:01.768960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.768730    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:44:01.769108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.768812    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:44:01.769108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.769026    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:44:01.772622 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.772589    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"]
Oct 30 12:44:01.793343 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.793285    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:01.807839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.807799    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.808067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.807946    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rm8\" (UniqueName: \"kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.808067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.808027    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.909237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.909134    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rm8\" (UniqueName: \"kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.909436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.909236    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.909436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.909358    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.910090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.910065    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.915984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.915962    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:01.933139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:01.933108    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rm8\" (UniqueName: \"kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8\") pod \"maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:02.071864 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:02.071815    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:02.150183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:02.149931    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:02.337626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:02.337583    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3579ae66-6544-4539-9ede-0ee800706f99" path="/var/lib/kubelet/pods/3579ae66-6544-4539-9ede-0ee800706f99/volumes"
Oct 30 12:44:02.379503 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:02.377947    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bb4295_71a8_4530_9d79_df11becb3c10.slice/crio-c17e815853e1df064ad589efa10c94d74a4ee88735f8b90ce2bf24b1cfd450af WatchSource:0}: Error finding container c17e815853e1df064ad589efa10c94d74a4ee88735f8b90ce2bf24b1cfd450af: Status 404 returned error can't find the container with id c17e815853e1df064ad589efa10c94d74a4ee88735f8b90ce2bf24b1cfd450af
Oct 30 12:44:02.381359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:02.381294    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"]
Oct 30 12:44:02.403148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:02.403108    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq" event={"ID":"59bb4295-71a8-4530-9d79-df11becb3c10","Type":"ContainerStarted","Data":"c17e815853e1df064ad589efa10c94d74a4ee88735f8b90ce2bf24b1cfd450af"}
Oct 30 12:44:03.414280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:03.414243    5060 generic.go:356] "Generic (PLEG): container finished" podID="59bb4295-71a8-4530-9d79-df11becb3c10" containerID="56a4c4d178cbeea2c41676552383ef148681c067c0f8436446e6e8a01db86bb2" exitCode=0
Oct 30 12:44:03.414684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:03.414307    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq" event={"ID":"59bb4295-71a8-4530-9d79-df11becb3c10","Type":"ContainerDied","Data":"56a4c4d178cbeea2c41676552383ef148681c067c0f8436446e6e8a01db86bb2"}
Oct 30 12:44:03.511792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:03.511755    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:03.904212 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:03.904184    5060 scope.go:122] "RemoveContainer" containerID="4bad47c9baaaed5aca240f0003f7d63db485ca009272a0d8e202c29e5d96288f"
Oct 30 12:44:03.915531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:03.915499    5060 scope.go:122] "RemoveContainer" containerID="58acc6e1a01be75e8bc095b028d86f6aff1a1d64c084e95cad421a8961c37546"
Oct 30 12:44:04.016049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.016007    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerName="payload-processing" containerID="cri-o://eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674" gracePeriod=25
Oct 30 12:44:04.607860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.607828    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:04.766074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.766041    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca\") pod \"59bb4295-71a8-4530-9d79-df11becb3c10\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") "
Oct 30 12:44:04.766304 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.766102    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8\" (UniqueName: \"kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8\") pod \"59bb4295-71a8-4530-9d79-df11becb3c10\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") "
Oct 30 12:44:04.766399 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.766384    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca\") pod \"59bb4295-71a8-4530-9d79-df11becb3c10\" (UID: \"59bb4295-71a8-4530-9d79-df11becb3c10\") "
Oct 30 12:44:04.766743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.766721    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca" pod "59bb4295-71a8-4530-9d79-df11becb3c10" (UID: "59bb4295-71a8-4530-9d79-df11becb3c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:04.766863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.766839    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:04.768421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.768398    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8" pod "59bb4295-71a8-4530-9d79-df11becb3c10" (UID: "59bb4295-71a8-4530-9d79-df11becb3c10"). InnerVolumeSpecName "kube-api-access-q8rm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:04.768421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.768410    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca" pod "59bb4295-71a8-4530-9d79-df11becb3c10" (UID: "59bb4295-71a8-4530-9d79-df11becb3c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:04.868080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.868048    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4295-71a8-4530-9d79-df11becb3c10-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:04.868080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:04.868076    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8rm8\" (UniqueName: \"kubernetes.io/projected/59bb4295-71a8-4530-9d79-df11becb3c10-kube-api-access-q8rm8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:05.229020 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:05.228970    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:05.436303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:05.436274    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"
Oct 30 12:44:05.436517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:05.436270    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq" event={"ID":"59bb4295-71a8-4530-9d79-df11becb3c10","Type":"ContainerDied","Data":"c17e815853e1df064ad589efa10c94d74a4ee88735f8b90ce2bf24b1cfd450af"}
Oct 30 12:44:05.436517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:05.436422    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17e815853e1df064ad589efa10c94d74a4ee88735f8b90ce2bf24b1cfd450af"
Oct 30 12:44:06.134057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.134014    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"]
Oct 30 12:44:06.134701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.134684    5060 cpu_manager.go:414] "containerMap: removing container" podUID="59bb4295-71a8-4530-9d79-df11becb3c10" containerName="revoke-keys"
Oct 30 12:44:06.134701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.134701    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bb4295-71a8-4530-9d79-df11becb3c10" containerName="revoke-keys"
Oct 30 12:44:06.134827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.134822    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bb4295-71a8-4530-9d79-df11becb3c10" containerName="revoke-keys"
Oct 30 12:44:06.138408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.138390    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.156238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.156209    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"]
Oct 30 12:44:06.163558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.163531    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:44:06.163748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.163681    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:44:06.163863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.163844    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:44:06.165678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.165656    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:44:06.286778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.286692    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44v4g\" (UniqueName: \"kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.286778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.286734    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.287002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.286914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.388105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.388020    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.388301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.388120    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44v4g\" (UniqueName: \"kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.388301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.388159    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.388901 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.388870    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.391080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.391051    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.412341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.412316    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44v4g\" (UniqueName: \"kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g\") pod \"maas-api-revoke-sub-fff61e17-qdn47\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.452527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.452500    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:06.773471 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:06.773429    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b66150_d4bc_424f_81e1_c98c768068e6.slice/crio-0df373fef10da54ad6bce1259c87b55384ff35f5d86db545ea00dcfdf946c82a WatchSource:0}: Error finding container 0df373fef10da54ad6bce1259c87b55384ff35f5d86db545ea00dcfdf946c82a: Status 404 returned error can't find the container with id 0df373fef10da54ad6bce1259c87b55384ff35f5d86db545ea00dcfdf946c82a
Oct 30 12:44:06.776476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:06.776403    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"]
Oct 30 12:44:07.039747 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:07.039669    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:07.454205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:07.454171    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47" event={"ID":"21b66150-d4bc-424f-81e1-c98c768068e6","Type":"ContainerStarted","Data":"bb59719d484a7d8141839017689ace756b4e21cd32393e40d40f7bca7ce1f4f5"}
Oct 30 12:44:07.454703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:07.454233    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47" event={"ID":"21b66150-d4bc-424f-81e1-c98c768068e6","Type":"ContainerStarted","Data":"0df373fef10da54ad6bce1259c87b55384ff35f5d86db545ea00dcfdf946c82a"}
Oct 30 12:44:07.500877 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:07.500811    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47" podStartSLOduration=1.5007915299999999 podStartE2EDuration="1.50079153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:07.49430536 +0000 UTC m=+4451.289189854" watchObservedRunningTime="2026-10-30 12:44:07.50079153 +0000 UTC m=+4451.295676027"
Oct 30 12:44:08.323676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.323635    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:08.464682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.464650    5060 generic.go:356] "Generic (PLEG): container finished" podID="21b66150-d4bc-424f-81e1-c98c768068e6" containerID="bb59719d484a7d8141839017689ace756b4e21cd32393e40d40f7bca7ce1f4f5" exitCode=0
Oct 30 12:44:08.465074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.464728    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47" event={"ID":"21b66150-d4bc-424f-81e1-c98c768068e6","Type":"ContainerDied","Data":"bb59719d484a7d8141839017689ace756b4e21cd32393e40d40f7bca7ce1f4f5"}
Oct 30 12:44:08.479004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.478974    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:08.633236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.633153    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"]
Oct 30 12:44:08.633726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.633693    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy" containerID="cri-o://575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244" gracePeriod=30
Oct 30 12:44:08.693227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.693169    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:08.834524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:08.831524    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:09.227712 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.227679    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"]
Oct 30 12:44:09.232160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.232140    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.249505 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.249481    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"]
Oct 30 12:44:09.339724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.339689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7v2l\" (UniqueName: \"kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.339903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.339742    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.339903 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.339838    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.441512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.441447    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.441771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.441749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7v2l\" (UniqueName: \"kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.441894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.441813    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.442494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.442441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.444122 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.444096    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.465468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.465436    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7v2l\" (UniqueName: \"kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l\") pod \"maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.546707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.546679    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:09.658784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.658758    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:09.748923 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.748894    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g\" (UniqueName: \"kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g\") pod \"21b66150-d4bc-424f-81e1-c98c768068e6\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") "
Oct 30 12:44:09.749090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.748942    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca\") pod \"21b66150-d4bc-424f-81e1-c98c768068e6\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") "
Oct 30 12:44:09.749090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.749020    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca\") pod \"21b66150-d4bc-424f-81e1-c98c768068e6\" (UID: \"21b66150-d4bc-424f-81e1-c98c768068e6\") "
Oct 30 12:44:09.749333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.749310    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca" pod "21b66150-d4bc-424f-81e1-c98c768068e6" (UID: "21b66150-d4bc-424f-81e1-c98c768068e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:09.751200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.751176    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g" pod "21b66150-d4bc-424f-81e1-c98c768068e6" (UID: "21b66150-d4bc-424f-81e1-c98c768068e6"). InnerVolumeSpecName "kube-api-access-44v4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:09.751279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.751199    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca" pod "21b66150-d4bc-424f-81e1-c98c768068e6" (UID: "21b66150-d4bc-424f-81e1-c98c768068e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:09.850999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.850921    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:09.850999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.850949    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b66150-d4bc-424f-81e1-c98c768068e6-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:09.850999 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.850958    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44v4g\" (UniqueName: \"kubernetes.io/projected/21b66150-d4bc-424f-81e1-c98c768068e6-kube-api-access-44v4g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:09.875105 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:09.875079    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5349a8e3_1450_4b99_b7c2_2b0ebe10afd3.slice/crio-a0be04e5aec3306a6456ef6e801f11a470093053a2f2f31309c3e677797c85ea WatchSource:0}: Error finding container a0be04e5aec3306a6456ef6e801f11a470093053a2f2f31309c3e677797c85ea: Status 404 returned error can't find the container with id a0be04e5aec3306a6456ef6e801f11a470093053a2f2f31309c3e677797c85ea
Oct 30 12:44:09.877793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:09.877762    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"]
Oct 30 12:44:10.486013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.485977    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd" event={"ID":"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3","Type":"ContainerStarted","Data":"e4c4db602457d9bc8263d0d52ff87ae49cc554c4990cd9bfaf8a6e2015fd84a0"}
Oct 30 12:44:10.486500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.486036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd" event={"ID":"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3","Type":"ContainerStarted","Data":"a0be04e5aec3306a6456ef6e801f11a470093053a2f2f31309c3e677797c85ea"}
Oct 30 12:44:10.487677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.487649    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47"
Oct 30 12:44:10.487764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.487670    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fff61e17-qdn47" event={"ID":"21b66150-d4bc-424f-81e1-c98c768068e6","Type":"ContainerDied","Data":"0df373fef10da54ad6bce1259c87b55384ff35f5d86db545ea00dcfdf946c82a"}
Oct 30 12:44:10.487764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.487713    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0df373fef10da54ad6bce1259c87b55384ff35f5d86db545ea00dcfdf946c82a"
Oct 30 12:44:10.531842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.531775    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd" podStartSLOduration=1.531758434 podStartE2EDuration="1.531758434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:10.52671091 +0000 UTC m=+4454.321595405" watchObservedRunningTime="2026-10-30 12:44:10.531758434 +0000 UTC m=+4454.326642931"
Oct 30 12:44:10.751294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:10.746220    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:11.503526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:11.503487    5060 generic.go:356] "Generic (PLEG): container finished" podID="5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" containerID="e4c4db602457d9bc8263d0d52ff87ae49cc554c4990cd9bfaf8a6e2015fd84a0" exitCode=0
Oct 30 12:44:11.504023 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:11.503503    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd" event={"ID":"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3","Type":"ContainerDied","Data":"e4c4db602457d9bc8263d0d52ff87ae49cc554c4990cd9bfaf8a6e2015fd84a0"}
Oct 30 12:44:11.768797 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:11.768704    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:12.290180 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.290142    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"]
Oct 30 12:44:12.326029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.325999    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"]
Oct 30 12:44:12.472823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.472778    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:12.618874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.618807    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:12.710157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.710125    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:12.801274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.801238    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l\" (UniqueName: \"kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l\") pod \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") "
Oct 30 12:44:12.801510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.801291    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca\") pod \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") "
Oct 30 12:44:12.801510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.801402    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca\") pod \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\" (UID: \"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3\") "
Oct 30 12:44:12.801713 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.801689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca" pod "5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" (UID: "5349a8e3-1450-4b99-b7c2-2b0ebe10afd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:12.802009 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.801988    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:12.803831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.803809    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca" pod "5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" (UID: "5349a8e3-1450-4b99-b7c2-2b0ebe10afd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:12.804119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.804102    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l" pod "5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" (UID: "5349a8e3-1450-4b99-b7c2-2b0ebe10afd3"). InnerVolumeSpecName "kube-api-access-k7v2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:12.903435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.903351    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7v2l\" (UniqueName: \"kubernetes.io/projected/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-kube-api-access-k7v2l\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:12.903435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:12.903384    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:13.524874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:13.524833    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd" event={"ID":"5349a8e3-1450-4b99-b7c2-2b0ebe10afd3","Type":"ContainerDied","Data":"a0be04e5aec3306a6456ef6e801f11a470093053a2f2f31309c3e677797c85ea"}
Oct 30 12:44:13.525064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:13.524900    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0be04e5aec3306a6456ef6e801f11a470093053a2f2f31309c3e677797c85ea"
Oct 30 12:44:13.525064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:13.524898    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"
Oct 30 12:44:13.945954 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:13.945919    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:44:14.022845 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.022808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.022873    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.022973    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023003    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023053    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023119    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.023261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023240    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023275    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023275    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.023433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023355    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token\") pod \"1b033509-5c13-468f-8c29-28ff246b975f\" (UID: \"1b033509-5c13-468f-8c29-28ff246b975f\") "
Oct 30 12:44:14.023993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023812    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.023993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023858    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:14.023993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023878    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.023993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.023901    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.024193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.024024    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.026834 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.026803    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:14.026953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.026859    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "kube-api-access-gqz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:14.027092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.027072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.027303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.027286    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo" pod "1b033509-5c13-468f-8c29-28ff246b975f" (UID: "1b033509-5c13-468f-8c29-28ff246b975f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125417    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqz95\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-kube-api-access-gqz95\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125478    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b033509-5c13-468f-8c29-28ff246b975f-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125490    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125498    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b033509-5c13-468f-8c29-28ff246b975f-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125506    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b033509-5c13-468f-8c29-28ff246b975f-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125513    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.125526 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.125523    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b033509-5c13-468f-8c29-28ff246b975f-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.248024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.247980    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:14.312263 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.312211    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" podUID="8a490272-861e-43c2-bd2e-7d4c6b415c39" containerName="main" containerID="cri-o://2094b4465f160ebac725d2008dfec123b8a17d2f4100e8d59f3a0f76a5df33b8" gracePeriod=45
Oct 30 12:44:14.385229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.385130    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:14.542297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.542264    5060 generic.go:356] "Generic (PLEG): container finished" podID="8a490272-861e-43c2-bd2e-7d4c6b415c39" containerID="2094b4465f160ebac725d2008dfec123b8a17d2f4100e8d59f3a0f76a5df33b8" exitCode=0
Oct 30 12:44:14.542559 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.542343    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" event={"ID":"8a490272-861e-43c2-bd2e-7d4c6b415c39","Type":"ContainerDied","Data":"2094b4465f160ebac725d2008dfec123b8a17d2f4100e8d59f3a0f76a5df33b8"}
Oct 30 12:44:14.544395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.544369    5060 generic.go:356] "Generic (PLEG): container finished" podID="1b033509-5c13-468f-8c29-28ff246b975f" containerID="575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244" exitCode=0
Oct 30 12:44:14.544549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.544475    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" event={"ID":"1b033509-5c13-468f-8c29-28ff246b975f","Type":"ContainerDied","Data":"575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244"}
Oct 30 12:44:14.544549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.544511    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5" event={"ID":"1b033509-5c13-468f-8c29-28ff246b975f","Type":"ContainerDied","Data":"f93bce09bc8b8b187f7079be65c48126f13296df5104617fa279ce1cfbd304a0"}
Oct 30 12:44:14.544549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.544530    5060 scope.go:122] "RemoveContainer" containerID="575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244"
Oct 30 12:44:14.544768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.544657    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"
Oct 30 12:44:14.557738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.557710    5060 scope.go:122] "RemoveContainer" containerID="1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"
Oct 30 12:44:14.569236 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.569216    5060 scope.go:122] "RemoveContainer" containerID="575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244"
Oct 30 12:44:14.569599 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:14.569564    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244\": container with ID starting with 575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244 not found: ID does not exist" containerID="575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244"
Oct 30 12:44:14.569692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.569609    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244"} err="failed to get container status \"575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244\": rpc error: code = NotFound desc = could not find container \"575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244\": container with ID starting with 575f27a8df1eab3757fae6a61cb78004c095c72ff08d84b9c915e49e3fdf8244 not found: ID does not exist"
Oct 30 12:44:14.569692 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.569624    5060 scope.go:122] "RemoveContainer" containerID="1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"
Oct 30 12:44:14.569929 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:14.569894    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89\": container with ID starting with 1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89 not found: ID does not exist" containerID="1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"
Oct 30 12:44:14.569980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.569941    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89"} err="failed to get container status \"1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89\": rpc error: code = NotFound desc = could not find container \"1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89\": container with ID starting with 1aef298aae202fbcc40f0d5ca89a800916096c9f51902dbe3744ffe942e87f89 not found: ID does not exist"
Oct 30 12:44:14.587787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.587762    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:44:14.599955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.599931    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"]
Oct 30 12:44:14.614907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.614874    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-67e819-openshift-default-5cc7f5fcbf-9l6c5"]
Oct 30 12:44:14.631674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.631647    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir\") pod \"8a490272-861e-43c2-bd2e-7d4c6b415c39\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") "
Oct 30 12:44:14.631835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.631685    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt\" (UniqueName: \"kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt\") pod \"8a490272-861e-43c2-bd2e-7d4c6b415c39\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") "
Oct 30 12:44:14.631885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.631861    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs\") pod \"8a490272-861e-43c2-bd2e-7d4c6b415c39\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") "
Oct 30 12:44:14.631921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.631910    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm\") pod \"8a490272-861e-43c2-bd2e-7d4c6b415c39\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") "
Oct 30 12:44:14.632019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.631908    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir" pod "8a490272-861e-43c2-bd2e-7d4c6b415c39" (UID: "8a490272-861e-43c2-bd2e-7d4c6b415c39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.632061 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632003    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache\") pod \"8a490272-861e-43c2-bd2e-7d4c6b415c39\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") "
Oct 30 12:44:14.632147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632115    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home\") pod \"8a490272-861e-43c2-bd2e-7d4c6b415c39\" (UID: \"8a490272-861e-43c2-bd2e-7d4c6b415c39\") "
Oct 30 12:44:14.632245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632208    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache" pod "8a490272-861e-43c2-bd2e-7d4c6b415c39" (UID: "8a490272-861e-43c2-bd2e-7d4c6b415c39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.632407 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632372    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home" pod "8a490272-861e-43c2-bd2e-7d4c6b415c39" (UID: "8a490272-861e-43c2-bd2e-7d4c6b415c39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.632666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632649    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.632757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632671    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.632757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.632686    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.634115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.634097    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt" pod "8a490272-861e-43c2-bd2e-7d4c6b415c39" (UID: "8a490272-861e-43c2-bd2e-7d4c6b415c39"). InnerVolumeSpecName "kube-api-access-n4pwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:14.634517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.634501    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm" pod "8a490272-861e-43c2-bd2e-7d4c6b415c39" (UID: "8a490272-861e-43c2-bd2e-7d4c6b415c39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:14.634576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.634522    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs" pod "8a490272-861e-43c2-bd2e-7d4c6b415c39" (UID: "8a490272-861e-43c2-bd2e-7d4c6b415c39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:14.733973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.733934    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4pwt\" (UniqueName: \"kubernetes.io/projected/8a490272-861e-43c2-bd2e-7d4c6b415c39-kube-api-access-n4pwt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.733973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.733971    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a490272-861e-43c2-bd2e-7d4c6b415c39-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:14.734253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:14.733983    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a490272-861e-43c2-bd2e-7d4c6b415c39-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:15.553737 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.553708    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"
Oct 30 12:44:15.554429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.553743    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" event={"ID":"8a490272-861e-43c2-bd2e-7d4c6b415c39","Type":"ContainerDied","Data":"89014713eb3ef6de669c1acd62636d11fe00178af43d48d88629d87433fe35ab"}
Oct 30 12:44:15.554429 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.553792    5060 scope.go:122] "RemoveContainer" containerID="2094b4465f160ebac725d2008dfec123b8a17d2f4100e8d59f3a0f76a5df33b8"
Oct 30 12:44:15.558509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.558480    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm" podUID="8a490272-861e-43c2-bd2e-7d4c6b415c39" containerName="main" probeResult="failure" output="Get \"https://10.132.1.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:44:15.609370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.609326    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"]
Oct 30 12:44:15.615885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.615851    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f1b14f/sub-test-model-f1b14f-kserve-6996df5589-s68hm"]
Oct 30 12:44:15.999980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:15.999944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:16.337777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.337696    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b033509-5c13-468f-8c29-28ff246b975f" path="/var/lib/kubelet/pods/1b033509-5c13-468f-8c29-28ff246b975f/volumes"
Oct 30 12:44:16.338156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.338141    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a490272-861e-43c2-bd2e-7d4c6b415c39" path="/var/lib/kubelet/pods/8a490272-861e-43c2-bd2e-7d4c6b415c39/volumes"
Oct 30 12:44:16.447117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.447071    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd" podUID="2c8db401-80b6-4da8-8a88-1e183e80b2be" containerName="main" containerID="cri-o://10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c" gracePeriod=45
Oct 30 12:44:16.732717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.732688    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:44:16.860314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860285    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm\") pod \"2c8db401-80b6-4da8-8a88-1e183e80b2be\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") "
Oct 30 12:44:16.860314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860320    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir\") pod \"2c8db401-80b6-4da8-8a88-1e183e80b2be\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") "
Oct 30 12:44:16.860594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860415    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs\") pod \"2c8db401-80b6-4da8-8a88-1e183e80b2be\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") "
Oct 30 12:44:16.860594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860501    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt\" (UniqueName: \"kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt\") pod \"2c8db401-80b6-4da8-8a88-1e183e80b2be\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") "
Oct 30 12:44:16.860594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860526    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home\") pod \"2c8db401-80b6-4da8-8a88-1e183e80b2be\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") "
Oct 30 12:44:16.860795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860658    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache\") pod \"2c8db401-80b6-4da8-8a88-1e183e80b2be\" (UID: \"2c8db401-80b6-4da8-8a88-1e183e80b2be\") "
Oct 30 12:44:16.860795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir" pod "2c8db401-80b6-4da8-8a88-1e183e80b2be" (UID: "2c8db401-80b6-4da8-8a88-1e183e80b2be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:16.860907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860860    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache" pod "2c8db401-80b6-4da8-8a88-1e183e80b2be" (UID: "2c8db401-80b6-4da8-8a88-1e183e80b2be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:16.860907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.860877    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home" pod "2c8db401-80b6-4da8-8a88-1e183e80b2be" (UID: "2c8db401-80b6-4da8-8a88-1e183e80b2be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:16.861195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.861158    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:16.861195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.861183    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:16.861195 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.861192    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:16.862748 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.862714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm" pod "2c8db401-80b6-4da8-8a88-1e183e80b2be" (UID: "2c8db401-80b6-4da8-8a88-1e183e80b2be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:16.863203 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.863185    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs" pod "2c8db401-80b6-4da8-8a88-1e183e80b2be" (UID: "2c8db401-80b6-4da8-8a88-1e183e80b2be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:16.863269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.863218    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt" pod "2c8db401-80b6-4da8-8a88-1e183e80b2be" (UID: "2c8db401-80b6-4da8-8a88-1e183e80b2be"). InnerVolumeSpecName "kube-api-access-8hltt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:16.962406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.962373    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hltt\" (UniqueName: \"kubernetes.io/projected/2c8db401-80b6-4da8-8a88-1e183e80b2be-kube-api-access-8hltt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:16.962406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.962403    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8db401-80b6-4da8-8a88-1e183e80b2be-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:16.962650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:16.962414    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8db401-80b6-4da8-8a88-1e183e80b2be-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:17.033156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.033107    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"]
Oct 30 12:44:17.111727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.111685    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:17.290936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.290841    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerName="payload-processing" containerID="cri-o://935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc" gracePeriod=25
Oct 30 12:44:17.327140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.327087    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerName="payload-pre-processing" containerID="cri-o://057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4" gracePeriod=25
Oct 30 12:44:17.573939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.573850    5060 generic.go:356] "Generic (PLEG): container finished" podID="2c8db401-80b6-4da8-8a88-1e183e80b2be" containerID="10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c" exitCode=0
Oct 30 12:44:17.573939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.573911    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"
Oct 30 12:44:17.574171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.573941    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd" event={"ID":"2c8db401-80b6-4da8-8a88-1e183e80b2be","Type":"ContainerDied","Data":"10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c"}
Oct 30 12:44:17.574171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.573990    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd" event={"ID":"2c8db401-80b6-4da8-8a88-1e183e80b2be","Type":"ContainerDied","Data":"9662ca418277d2c73b3d0f7371f29740d15b5e02eee0c641f2246564b910c2f3"}
Oct 30 12:44:17.574171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.574009    5060 scope.go:122] "RemoveContainer" containerID="10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c"
Oct 30 12:44:17.588214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.588196    5060 scope.go:122] "RemoveContainer" containerID="10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c"
Oct 30 12:44:17.588555 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:17.588529    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c\": container with ID starting with 10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c not found: ID does not exist" containerID="10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c"
Oct 30 12:44:17.588640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.588570    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c"} err="failed to get container status \"10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c\": rpc error: code = NotFound desc = could not find container \"10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c\": container with ID starting with 10e4a8dc0e3518ed0b75e1e21dfe3deaab69da605f233a202a1c7d6ff9bf887c not found: ID does not exist"
Oct 30 12:44:17.628046 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.628014    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"]
Oct 30 12:44:17.632068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.632043    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4fff4f/sub-test-model-4fff4f-kserve-769df8f78f-hkmxd"]
Oct 30 12:44:17.764370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:17.764325    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:18.335793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:18.335758    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8db401-80b6-4da8-8a88-1e183e80b2be" path="/var/lib/kubelet/pods/2c8db401-80b6-4da8-8a88-1e183e80b2be/volumes"
Oct 30 12:44:18.817495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:18.817434    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:19.012776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.012736    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"]
Oct 30 12:44:19.013659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013638    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy"
Oct 30 12:44:19.013727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013662    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy"
Oct 30 12:44:19.013727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013692    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy"
Oct 30 12:44:19.013727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013700    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy"
Oct 30 12:44:19.013727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013721    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2c8db401-80b6-4da8-8a88-1e183e80b2be" containerName="main"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013728    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8db401-80b6-4da8-8a88-1e183e80b2be" containerName="main"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013757    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" containerName="revoke-keys"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013764    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" containerName="revoke-keys"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013773    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8a490272-861e-43c2-bd2e-7d4c6b415c39" containerName="main"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013780    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a490272-861e-43c2-bd2e-7d4c6b415c39" containerName="main"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013795    5060 cpu_manager.go:414] "containerMap: removing container" podUID="21b66150-d4bc-424f-81e1-c98c768068e6" containerName="revoke-keys"
Oct 30 12:44:19.013862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013802    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b66150-d4bc-424f-81e1-c98c768068e6" containerName="revoke-keys"
Oct 30 12:44:19.014099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013933    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy"
Oct 30 12:44:19.014099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013946    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8db401-80b6-4da8-8a88-1e183e80b2be" containerName="main"
Oct 30 12:44:19.014099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013962    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" containerName="revoke-keys"
Oct 30 12:44:19.014099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013972    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a490272-861e-43c2-bd2e-7d4c6b415c39" containerName="main"
Oct 30 12:44:19.014099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.013983    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b66150-d4bc-424f-81e1-c98c768068e6" containerName="revoke-keys"
Oct 30 12:44:19.019606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.019583    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.033827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.033795    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"]
Oct 30 12:44:19.039662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.039634    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:44:19.039857 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.039683    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:44:19.039976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.039747    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:44:19.047266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.047242    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:44:19.099501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.099364    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.099501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.099415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtp4g\" (UniqueName: \"kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.099501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.099498    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.201325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.201285    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.201585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.201335    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtp4g\" (UniqueName: \"kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.201585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.201392    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.202337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.202310    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.213968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.213920    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.232809 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.232768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtp4g\" (UniqueName: \"kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.306945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.306882    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:19.337002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.336969    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:19.686716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.686554    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"]
Oct 30 12:44:19.898945 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:19.898909    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:44:19.910396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.910351    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:19.910396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.910387    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:19.959470 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:19.959423    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:20.069388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.069353    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"]
Oct 30 12:44:20.073254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.073222    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4f7443-p9mqq"]
Oct 30 12:44:20.111141 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.110693    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:20.335777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.335692    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bb4295-71a8-4530-9d79-df11becb3c10" path="/var/lib/kubelet/pods/59bb4295-71a8-4530-9d79-df11becb3c10/volumes"
Oct 30 12:44:20.606446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.606356    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb" event={"ID":"c7ec97fc-7e5d-4019-9875-3543903400f5","Type":"ContainerStarted","Data":"bddfb5a1e241a24f620929061dacde6cf7d10be6750238609c91067006abdf67"}
Oct 30 12:44:20.606446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.606419    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb" event={"ID":"c7ec97fc-7e5d-4019-9875-3543903400f5","Type":"ContainerStarted","Data":"0f9388602ffb9bad00f3fd47e011588cbee53370ff5115b6d27ce93b82a55260"}
Oct 30 12:44:20.658563 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.658485    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb" podStartSLOduration=2.658436875 podStartE2EDuration="2.658436875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:20.652165252 +0000 UTC m=+4464.447049760" watchObservedRunningTime="2026-10-30 12:44:20.658436875 +0000 UTC m=+4464.453321370"
Oct 30 12:44:20.662257 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:20.662217    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:21.164756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:21.164663    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:21.363938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:21.363891    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:21.680158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:21.680002    5060 generic.go:356] "Generic (PLEG): container finished" podID="c7ec97fc-7e5d-4019-9875-3543903400f5" containerID="bddfb5a1e241a24f620929061dacde6cf7d10be6750238609c91067006abdf67" exitCode=0
Oct 30 12:44:21.680396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:21.680179    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb" event={"ID":"c7ec97fc-7e5d-4019-9875-3543903400f5","Type":"ContainerDied","Data":"bddfb5a1e241a24f620929061dacde6cf7d10be6750238609c91067006abdf67"}
Oct 30 12:44:21.768522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:21.768480    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:22.021524 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.021477    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:22.863891 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.863867    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:22.977465 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.977417    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca\") pod \"c7ec97fc-7e5d-4019-9875-3543903400f5\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") "
Oct 30 12:44:22.977638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.977551    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g\" (UniqueName: \"kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g\") pod \"c7ec97fc-7e5d-4019-9875-3543903400f5\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") "
Oct 30 12:44:22.977638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.977582    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca\") pod \"c7ec97fc-7e5d-4019-9875-3543903400f5\" (UID: \"c7ec97fc-7e5d-4019-9875-3543903400f5\") "
Oct 30 12:44:22.977982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.977959    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca" pod "c7ec97fc-7e5d-4019-9875-3543903400f5" (UID: "c7ec97fc-7e5d-4019-9875-3543903400f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:22.978104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.978078    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:22.980346 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.980322    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca" pod "c7ec97fc-7e5d-4019-9875-3543903400f5" (UID: "c7ec97fc-7e5d-4019-9875-3543903400f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:22.980445 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:22.980370    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g" pod "c7ec97fc-7e5d-4019-9875-3543903400f5" (UID: "c7ec97fc-7e5d-4019-9875-3543903400f5"). InnerVolumeSpecName "kube-api-access-qtp4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:23.079572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:23.079529    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtp4g\" (UniqueName: \"kubernetes.io/projected/c7ec97fc-7e5d-4019-9875-3543903400f5-kube-api-access-qtp4g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:23.079572 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:23.079570    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ec97fc-7e5d-4019-9875-3543903400f5-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:23.698467 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:23.698429    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"
Oct 30 12:44:23.698659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:23.698477    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb" event={"ID":"c7ec97fc-7e5d-4019-9875-3543903400f5","Type":"ContainerDied","Data":"0f9388602ffb9bad00f3fd47e011588cbee53370ff5115b6d27ce93b82a55260"}
Oct 30 12:44:23.698659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:23.698518    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9388602ffb9bad00f3fd47e011588cbee53370ff5115b6d27ce93b82a55260"
Oct 30 12:44:23.713254 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:23.713221    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:24.291389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.291324    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:44:24.393153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.393116    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb\" (UniqueName: \"kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb\") pod \"edd701a8-818f-456a-8c94-fc10bffa13cc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") "
Oct 30 12:44:24.393370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.393317    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume\") pod \"edd701a8-818f-456a-8c94-fc10bffa13cc\" (UID: \"edd701a8-818f-456a-8c94-fc10bffa13cc\") "
Oct 30 12:44:24.394615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.394574    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume" pod "edd701a8-818f-456a-8c94-fc10bffa13cc" (UID: "edd701a8-818f-456a-8c94-fc10bffa13cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:24.395846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.395809    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb" pod "edd701a8-818f-456a-8c94-fc10bffa13cc" (UID: "edd701a8-818f-456a-8c94-fc10bffa13cc"). InnerVolumeSpecName "kube-api-access-pmfqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:24.495285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.495247    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd701a8-818f-456a-8c94-fc10bffa13cc-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:24.495285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.495285    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfqb\" (UniqueName: \"kubernetes.io/projected/edd701a8-818f-456a-8c94-fc10bffa13cc-kube-api-access-pmfqb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:24.710777 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.710739    5060 generic.go:356] "Generic (PLEG): container finished" podID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerID="eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674" exitCode=0
Oct 30 12:44:24.710998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.710782    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"
Oct 30 12:44:24.710998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.710829    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" event={"ID":"edd701a8-818f-456a-8c94-fc10bffa13cc","Type":"ContainerDied","Data":"eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674"}
Oct 30 12:44:24.710998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.710887    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc" event={"ID":"edd701a8-818f-456a-8c94-fc10bffa13cc","Type":"ContainerDied","Data":"07c76c248a83fb921a96017799ac2cd6b7238c2dfcd7f67847b5c3aa216debcf"}
Oct 30 12:44:24.710998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.710909    5060 scope.go:122] "RemoveContainer" containerID="eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674"
Oct 30 12:44:24.722440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.722413    5060 scope.go:122] "RemoveContainer" containerID="eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674"
Oct 30 12:44:24.722740 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:24.722718    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674\": container with ID starting with eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674 not found: ID does not exist" containerID="eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674"
Oct 30 12:44:24.722811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.722748    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674"} err="failed to get container status \"eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674\": rpc error: code = NotFound desc = could not find container \"eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674\": container with ID starting with eaa8eedd70ad0d9d7ce193ceede14bf7cb69f18273e159ece71b493e2c3b4674 not found: ID does not exist"
Oct 30 12:44:24.766618 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.766584    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"]
Oct 30 12:44:24.778482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:24.778438    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-67e819-5ffb9f9dd4-n94xc"]
Oct 30 12:44:25.471718 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:25.471532    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:26.338678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:26.338649    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" path="/var/lib/kubelet/pods/edd701a8-818f-456a-8c94-fc10bffa13cc/volumes"
Oct 30 12:44:27.043397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.043364    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:27.206297 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.206247    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:27.351104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.351005    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"]
Oct 30 12:44:27.351377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.351346    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy" containerID="cri-o://b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e" gracePeriod=30
Oct 30 12:44:27.365221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.365184    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:27.520478 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.519118    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:27.935829 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.935794    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"]
Oct 30 12:44:27.936377 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936360    5060 cpu_manager.go:414] "containerMap: removing container" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerName="payload-processing"
Oct 30 12:44:27.936431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936380    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerName="payload-processing"
Oct 30 12:44:27.936431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936402    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c7ec97fc-7e5d-4019-9875-3543903400f5" containerName="revoke-keys"
Oct 30 12:44:27.936431 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936408    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ec97fc-7e5d-4019-9875-3543903400f5" containerName="revoke-keys"
Oct 30 12:44:27.936556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936527    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b033509-5c13-468f-8c29-28ff246b975f" containerName="istio-proxy"
Oct 30 12:44:27.936556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936535    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd701a8-818f-456a-8c94-fc10bffa13cc" containerName="payload-processing"
Oct 30 12:44:27.936556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.936545    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ec97fc-7e5d-4019-9875-3543903400f5" containerName="revoke-keys"
Oct 30 12:44:27.941634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.941612    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:27.961002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.960976    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"]
Oct 30 12:44:27.968393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.968370    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:44:27.968531 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:27.968400    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:44:28.027991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.027961    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:44:28.027991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.027987    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:44:28.037688 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.037666    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.037821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.037717    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g6x\" (UniqueName: \"kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.037821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.037807    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.139546 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.139508    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.139959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.139565    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g6x\" (UniqueName: \"kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.139959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.139600    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.140313 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.140293    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.142125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.142106    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.163136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.163105    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g6x\" (UniqueName: \"kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x\") pod \"maas-api-revoke-keys-e2e-extmod-b-846099-mts2z\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.255353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.255326    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:28.565426 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:28.565395    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc9a16c_d7d2_4e80_b637_95f2a199b707.slice/crio-83683e6dde300dbb7b6f113915fc165d8c2bb3d41997b3e0144574f692e809c8 WatchSource:0}: Error finding container 83683e6dde300dbb7b6f113915fc165d8c2bb3d41997b3e0144574f692e809c8: Status 404 returned error can't find the container with id 83683e6dde300dbb7b6f113915fc165d8c2bb3d41997b3e0144574f692e809c8
Oct 30 12:44:28.567720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.567694    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"]
Oct 30 12:44:28.749469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:28.749410    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z" event={"ID":"dbc9a16c-d7d2-4e80-b637-95f2a199b707","Type":"ContainerStarted","Data":"83683e6dde300dbb7b6f113915fc165d8c2bb3d41997b3e0144574f692e809c8"}
Oct 30 12:44:29.336667 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.336627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:29.732047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.732012    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:29.760119 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.760087    5060 generic.go:356] "Generic (PLEG): container finished" podID="dbc9a16c-d7d2-4e80-b637-95f2a199b707" containerID="cb3eb524d1c502cede2c4233d565a42c6314b0f141197306ffd5596050afe90c" exitCode=0
Oct 30 12:44:29.760328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.760189    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z" event={"ID":"dbc9a16c-d7d2-4e80-b637-95f2a199b707","Type":"ContainerDied","Data":"cb3eb524d1c502cede2c4233d565a42c6314b0f141197306ffd5596050afe90c"}
Oct 30 12:44:29.873053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.873019    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:29.910219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.910182    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:29.910396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:29.910220    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:30.919818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.919789    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:30.976787 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.976755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca\") pod \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") "
Oct 30 12:44:30.976994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.976912    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca\") pod \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") "
Oct 30 12:44:30.976994 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.976978    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x\" (UniqueName: \"kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x\") pod \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\" (UID: \"dbc9a16c-d7d2-4e80-b637-95f2a199b707\") "
Oct 30 12:44:30.977219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.977149    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca" pod "dbc9a16c-d7d2-4e80-b637-95f2a199b707" (UID: "dbc9a16c-d7d2-4e80-b637-95f2a199b707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:30.977575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.977548    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:30.980010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.979856    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca" pod "dbc9a16c-d7d2-4e80-b637-95f2a199b707" (UID: "dbc9a16c-d7d2-4e80-b637-95f2a199b707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:30.980010 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:30.979985    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x" pod "dbc9a16c-d7d2-4e80-b637-95f2a199b707" (UID: "dbc9a16c-d7d2-4e80-b637-95f2a199b707"). InnerVolumeSpecName "kube-api-access-n8g6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:31.078755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:31.078672    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc9a16c-d7d2-4e80-b637-95f2a199b707-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:31.078755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:31.078709    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8g6x\" (UniqueName: \"kubernetes.io/projected/dbc9a16c-d7d2-4e80-b637-95f2a199b707-kube-api-access-n8g6x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:31.328426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:31.326792    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:31.778542 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:31.778513    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"
Oct 30 12:44:31.778730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:31.778505    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z" event={"ID":"dbc9a16c-d7d2-4e80-b637-95f2a199b707","Type":"ContainerDied","Data":"83683e6dde300dbb7b6f113915fc165d8c2bb3d41997b3e0144574f692e809c8"}
Oct 30 12:44:31.778730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:31.778652    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83683e6dde300dbb7b6f113915fc165d8c2bb3d41997b3e0144574f692e809c8"
Oct 30 12:44:32.059958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.059866    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" podUID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" containerName="main" containerID="cri-o://7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100" gracePeriod=45
Oct 30 12:44:32.479258 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.479234    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:44:32.600002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.599972    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home\") pod \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") "
Oct 30 12:44:32.600183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600014    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk\" (UniqueName: \"kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk\") pod \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") "
Oct 30 12:44:32.600183 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600057    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm\") pod \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") "
Oct 30 12:44:32.600276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600228    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir\") pod \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") "
Oct 30 12:44:32.600276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600239    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home" pod "9d3ce9be-cce5-466e-b3e9-f28e475fb26c" (UID: "9d3ce9be-cce5-466e-b3e9-f28e475fb26c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.600276 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600272    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs\") pod \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") "
Oct 30 12:44:32.600386 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache\") pod \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\" (UID: \"9d3ce9be-cce5-466e-b3e9-f28e475fb26c\") "
Oct 30 12:44:32.600749 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600583    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir" pod "9d3ce9be-cce5-466e-b3e9-f28e475fb26c" (UID: "9d3ce9be-cce5-466e-b3e9-f28e475fb26c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.601047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.600758    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache" pod "9d3ce9be-cce5-466e-b3e9-f28e475fb26c" (UID: "9d3ce9be-cce5-466e-b3e9-f28e475fb26c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.601117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.601078    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.601117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.601102    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.601315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.601117    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.603089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.603067    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm" pod "9d3ce9be-cce5-466e-b3e9-f28e475fb26c" (UID: "9d3ce9be-cce5-466e-b3e9-f28e475fb26c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.604217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.604193    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs" pod "9d3ce9be-cce5-466e-b3e9-f28e475fb26c" (UID: "9d3ce9be-cce5-466e-b3e9-f28e475fb26c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:32.604314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.604300    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk" pod "9d3ce9be-cce5-466e-b3e9-f28e475fb26c" (UID: "9d3ce9be-cce5-466e-b3e9-f28e475fb26c"). InnerVolumeSpecName "kube-api-access-lqrtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:32.661694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.661659    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:44:32.702828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.702800    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.702828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.702826    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrtk\" (UniqueName: \"kubernetes.io/projected/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-kube-api-access-lqrtk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.703013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.702835    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3ce9be-cce5-466e-b3e9-f28e475fb26c-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.788169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.788133    5060 generic.go:356] "Generic (PLEG): container finished" podID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" containerID="7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100" exitCode=0
Oct 30 12:44:32.788373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.788182    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"
Oct 30 12:44:32.788373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.788222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" event={"ID":"9d3ce9be-cce5-466e-b3e9-f28e475fb26c","Type":"ContainerDied","Data":"7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100"}
Oct 30 12:44:32.788373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.788276    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" event={"ID":"9d3ce9be-cce5-466e-b3e9-f28e475fb26c","Type":"ContainerDied","Data":"ea2f76d78291a62f3316f606b9b9b3a4117e02c59766ae3c7cf5ea79011e15f6"}
Oct 30 12:44:32.788373 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.788301    5060 scope.go:122] "RemoveContainer" containerID="7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100"
Oct 30 12:44:32.791335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.791314    5060 generic.go:356] "Generic (PLEG): container finished" podID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerID="b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e" exitCode=0
Oct 30 12:44:32.791495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.791360    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" event={"ID":"77b6f25c-bf59-42b1-9d3e-4251827e7a65","Type":"ContainerDied","Data":"b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e"}
Oct 30 12:44:32.791495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.791374    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"
Oct 30 12:44:32.791495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.791404    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf" event={"ID":"77b6f25c-bf59-42b1-9d3e-4251827e7a65","Type":"ContainerDied","Data":"083999c0658043b8cd8ea9307306512dba169fdacd89e2dbb9ad305564d4949a"}
Oct 30 12:44:32.802736 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.802717    5060 scope.go:122] "RemoveContainer" containerID="7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100"
Oct 30 12:44:32.803035 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:32.803012    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100\": container with ID starting with 7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100 not found: ID does not exist" containerID="7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100"
Oct 30 12:44:32.803196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803167    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100"} err="failed to get container status \"7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100\": rpc error: code = NotFound desc = could not find container \"7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100\": container with ID starting with 7a8a01c8cd78e27bbfe1ae8b9198c17659ed7ffcdc6f1a45852d6e08da90c100 not found: ID does not exist"
Oct 30 12:44:32.803196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803197    5060 scope.go:122] "RemoveContainer" containerID="b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e"
Oct 30 12:44:32.803388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803267    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.803388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803309    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.803388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803345    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.803598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803410    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.803598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803444    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.803716 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.803987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.803958    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.804113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.804012    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:32.804113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.804065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.804250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.804128    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.804250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.804161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo\") pod \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\" (UID: \"77b6f25c-bf59-42b1-9d3e-4251827e7a65\") "
Oct 30 12:44:32.804652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.804628    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.804741 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.804655    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.805773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.805751    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.805911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.805869    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.806134 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.806086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.806261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.806216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:32.806473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.806426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "kube-api-access-842mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:32.807004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.806974    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:32.815931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.815894    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo" pod "77b6f25c-bf59-42b1-9d3e-4251827e7a65" (UID: "77b6f25c-bf59-42b1-9d3e-4251827e7a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:44:32.824031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.824001    5060 scope.go:122] "RemoveContainer" containerID="01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"
Oct 30 12:44:32.834137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.834119    5060 scope.go:122] "RemoveContainer" containerID="b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e"
Oct 30 12:44:32.834434 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:32.834415    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e\": container with ID starting with b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e not found: ID does not exist" containerID="b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e"
Oct 30 12:44:32.834537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.834446    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e"} err="failed to get container status \"b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e\": rpc error: code = NotFound desc = could not find container \"b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e\": container with ID starting with b3ed2d1ce4b32ce2c47f5a923aa54e714c03c4a9051210399880a5a56ae4c28e not found: ID does not exist"
Oct 30 12:44:32.834537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.834485    5060 scope.go:122] "RemoveContainer" containerID="01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"
Oct 30 12:44:32.834772 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:32.834755    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe\": container with ID starting with 01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe not found: ID does not exist" containerID="01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"
Oct 30 12:44:32.834837 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.834780    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe"} err="failed to get container status \"01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe\": rpc error: code = NotFound desc = could not find container \"01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe\": container with ID starting with 01535fb3d22ee8b9fc5b33d035c9de7b7ed14f99aca78d65b8c805dd30068fbe not found: ID does not exist"
Oct 30 12:44:32.847311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.847288    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"]
Oct 30 12:44:32.854440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.854419    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx"]
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906040    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906068    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906083    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842mq\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-kube-api-access-842mq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906094    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b6f25c-bf59-42b1-9d3e-4251827e7a65-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906107    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906126    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:32.908550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:32.906139    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b6f25c-bf59-42b1-9d3e-4251827e7a65-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:33.000989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.000953    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:33.164539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.154193    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"]
Oct 30 12:44:33.164539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.162428    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-x2trf"]
Oct 30 12:44:33.395765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.395735    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:33.406575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.406544    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-0c1869/rate-test-model-0c1869-kserve-6fdf86fb45-8c8tx" podUID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.168:8000/ready\": context deadline exceeded"
Oct 30 12:44:33.542039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.542004    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:33.636575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.636527    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"]
Oct 30 12:44:33.640439 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.640413    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bae155-pgfwd"]
Oct 30 12:44:33.685983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:33.685948    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:34.125771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.125715    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:34.229789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.229759    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"]
Oct 30 12:44:34.233378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.233354    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0c1869-c6rsb"]
Oct 30 12:44:34.274616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.274566    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:34.336367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.336335    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5349a8e3-1450-4b99-b7c2-2b0ebe10afd3" path="/var/lib/kubelet/pods/5349a8e3-1450-4b99-b7c2-2b0ebe10afd3/volumes"
Oct 30 12:44:34.336911 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.336895    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" path="/var/lib/kubelet/pods/77b6f25c-bf59-42b1-9d3e-4251827e7a65/volumes"
Oct 30 12:44:34.337425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.337406    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" path="/var/lib/kubelet/pods/9d3ce9be-cce5-466e-b3e9-f28e475fb26c/volumes"
Oct 30 12:44:34.338303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.338286    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ec97fc-7e5d-4019-9875-3543903400f5" path="/var/lib/kubelet/pods/c7ec97fc-7e5d-4019-9875-3543903400f5/volumes"
Oct 30 12:44:34.772918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:34.772884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:36.160925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:36.160890    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:37.618548 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.618517    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:44:37.650339 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.650315    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:44:37.765028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.764995    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m\" (UniqueName: \"kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m\") pod \"e34ea3a0-dffb-41c5-82f1-ba407b590721\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") "
Oct 30 12:44:37.765218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.765049    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume\") pod \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") "
Oct 30 12:44:37.765218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.765088    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume\") pod \"e34ea3a0-dffb-41c5-82f1-ba407b590721\" (UID: \"e34ea3a0-dffb-41c5-82f1-ba407b590721\") "
Oct 30 12:44:37.765218 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.765197    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f\" (UniqueName: \"kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f\") pod \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\" (UID: \"18661ff5-1c6a-4e60-adfe-e036eb1ad34d\") "
Oct 30 12:44:37.765589 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.765558    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume" pod "e34ea3a0-dffb-41c5-82f1-ba407b590721" (UID: "e34ea3a0-dffb-41c5-82f1-ba407b590721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:37.765687 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.765558    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume" pod "18661ff5-1c6a-4e60-adfe-e036eb1ad34d" (UID: "18661ff5-1c6a-4e60-adfe-e036eb1ad34d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:37.767298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.767277    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m" pod "e34ea3a0-dffb-41c5-82f1-ba407b590721" (UID: "e34ea3a0-dffb-41c5-82f1-ba407b590721"). InnerVolumeSpecName "kube-api-access-xmr4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:37.767387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.767348    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f" pod "18661ff5-1c6a-4e60-adfe-e036eb1ad34d" (UID: "18661ff5-1c6a-4e60-adfe-e036eb1ad34d"). InnerVolumeSpecName "kube-api-access-l2s2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:37.838155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.838078    5060 generic.go:356] "Generic (PLEG): container finished" podID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerID="935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc" exitCode=0
Oct 30 12:44:37.838155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.838125    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"
Oct 30 12:44:37.838359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.838157    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" event={"ID":"e34ea3a0-dffb-41c5-82f1-ba407b590721","Type":"ContainerDied","Data":"935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc"}
Oct 30 12:44:37.838359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.838199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf" event={"ID":"e34ea3a0-dffb-41c5-82f1-ba407b590721","Type":"ContainerDied","Data":"83a3ec0dc00d6fa4c79d0ddfc22e70f510ad6e1d4f07e64d15c32c704a152268"}
Oct 30 12:44:37.838359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.838219    5060 scope.go:122] "RemoveContainer" containerID="935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc"
Oct 30 12:44:37.839879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.839858    5060 generic.go:356] "Generic (PLEG): container finished" podID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerID="057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4" exitCode=0
Oct 30 12:44:37.840012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.839890    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" event={"ID":"18661ff5-1c6a-4e60-adfe-e036eb1ad34d","Type":"ContainerDied","Data":"057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4"}
Oct 30 12:44:37.840012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.839895    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"
Oct 30 12:44:37.840012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.839922    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5" event={"ID":"18661ff5-1c6a-4e60-adfe-e036eb1ad34d","Type":"ContainerDied","Data":"3c40a7ebbdf33caa376d09703e2241d068c050eccb6d9472121ba64e00e35b3d"}
Oct 30 12:44:37.851185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.851168    5060 scope.go:122] "RemoveContainer" containerID="935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc"
Oct 30 12:44:37.851502 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:37.851483    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc\": container with ID starting with 935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc not found: ID does not exist" containerID="935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc"
Oct 30 12:44:37.851556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.851512    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc"} err="failed to get container status \"935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc\": rpc error: code = NotFound desc = could not find container \"935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc\": container with ID starting with 935723aa71cfd74be4628f38f2fa06835c8511756d902a84cdc3e9a5743fefcc not found: ID does not exist"
Oct 30 12:44:37.851556 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.851533    5060 scope.go:122] "RemoveContainer" containerID="057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4"
Oct 30 12:44:37.861628 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.861610    5060 scope.go:122] "RemoveContainer" containerID="057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4"
Oct 30 12:44:37.861867 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:37.861848    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4\": container with ID starting with 057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4 not found: ID does not exist" containerID="057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4"
Oct 30 12:44:37.861916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.861875    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4"} err="failed to get container status \"057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4\": rpc error: code = NotFound desc = could not find container \"057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4\": container with ID starting with 057e522c6623fbc30b86069fe8e8e3868bb041d1be14819d0f6c14d6b7b5e3f4 not found: ID does not exist"
Oct 30 12:44:37.866577 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.866558    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2s2f\" (UniqueName: \"kubernetes.io/projected/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-kube-api-access-l2s2f\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:37.866640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.866579    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmr4m\" (UniqueName: \"kubernetes.io/projected/e34ea3a0-dffb-41c5-82f1-ba407b590721-kube-api-access-xmr4m\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:37.866640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.866588    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18661ff5-1c6a-4e60-adfe-e036eb1ad34d-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:37.866640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.866596    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34ea3a0-dffb-41c5-82f1-ba407b590721-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:37.895648 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.895612    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"]
Oct 30 12:44:37.901586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.901562    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4f7443-7c58cf7bf5-cj6d5"]
Oct 30 12:44:37.934363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.934331    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"]
Oct 30 12:44:37.940874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.940852    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4f7443-f94dccdc6-26whf"]
Oct 30 12:44:37.945887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:37.945859    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:38.335039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:38.335010    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" path="/var/lib/kubelet/pods/18661ff5-1c6a-4e60-adfe-e036eb1ad34d/volumes"
Oct 30 12:44:38.335384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:38.335371    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" path="/var/lib/kubelet/pods/e34ea3a0-dffb-41c5-82f1-ba407b590721/volumes"
Oct 30 12:44:38.604570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:38.604479    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:38.774086 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:38.774051    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:39.812807 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:39.812772    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:41.058678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.058638    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:41.214962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.214912    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:41.410087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.409998    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"]
Oct 30 12:44:41.410642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.410604    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" containerID="cri-o://35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314" gracePeriod=30
Oct 30 12:44:41.512443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.512396    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:41.667190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.667087    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:41.977979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.977949    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"]
Oct 30 12:44:41.978528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978515    5060 cpu_manager.go:414] "containerMap: removing container" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy"
Oct 30 12:44:41.978597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978530    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy"
Oct 30 12:44:41.978597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978547    5060 cpu_manager.go:414] "containerMap: removing container" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerName="payload-pre-processing"
Oct 30 12:44:41.978597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978553    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerName="payload-pre-processing"
Oct 30 12:44:41.978597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978561    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerName="payload-processing"
Oct 30 12:44:41.978597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978570    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerName="payload-processing"
Oct 30 12:44:41.978597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978595    5060 cpu_manager.go:414] "containerMap: removing container" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978603    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978613    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" containerName="main"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978618    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" containerName="main"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978631    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dbc9a16c-d7d2-4e80-b637-95f2a199b707" containerName="revoke-keys"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978636    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc9a16c-d7d2-4e80-b637-95f2a199b707" containerName="revoke-keys"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978718    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978726    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="18661ff5-1c6a-4e60-adfe-e036eb1ad34d" containerName="payload-pre-processing"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978731    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34ea3a0-dffb-41c5-82f1-ba407b590721" containerName="payload-processing"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978740    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b6f25c-bf59-42b1-9d3e-4251827e7a65" containerName="istio-proxy"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978745    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3ce9be-cce5-466e-b3e9-f28e475fb26c" containerName="main"
Oct 30 12:44:41.978812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.978752    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc9a16c-d7d2-4e80-b637-95f2a199b707" containerName="revoke-keys"
Oct 30 12:44:41.983973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:41.983950    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.002843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.002814    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"]
Oct 30 12:44:42.009401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.009354    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:44:42.009401 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.009365    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:44:42.009705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.009461    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:44:42.009705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.009549    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:44:42.071158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.071123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:42.147919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.147882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnw4\" (UniqueName: \"kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.148155 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.147983    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.148241 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.148220    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.244049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.243958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:42.249270 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.249242    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.249403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.249297    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnw4\" (UniqueName: \"kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.249403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.249330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.249986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.249962    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.251846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.251819    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.296886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.296857    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnw4\" (UniqueName: \"kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.299594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.299567    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:42.466153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.466114    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:42.623627 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:42.623595    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f04e82e_8a4d_499a_93e8_3ebea7e4120d.slice/crio-1b03957cf815d9b9391d8d8fe15d39d4e02dd337dbdaa4a164edd545d1601fce WatchSource:0}: Error finding container 1b03957cf815d9b9391d8d8fe15d39d4e02dd337dbdaa4a164edd545d1601fce: Status 404 returned error can't find the container with id 1b03957cf815d9b9391d8d8fe15d39d4e02dd337dbdaa4a164edd545d1601fce
Oct 30 12:44:42.627053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.627025    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"]
Oct 30 12:44:42.882915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:42.882832    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts" event={"ID":"1f04e82e-8a4d-499a-93e8-3ebea7e4120d","Type":"ContainerStarted","Data":"1b03957cf815d9b9391d8d8fe15d39d4e02dd337dbdaa4a164edd545d1601fce"}
Oct 30 12:44:43.034098 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.034071    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"]
Oct 30 12:44:43.038139 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.038122    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.058142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.058113    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"]
Oct 30 12:44:43.064835 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.064810    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5j2f\" (UniqueName: \"kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.064964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.064852    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.064964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.064892    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.165793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.165699    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5j2f\" (UniqueName: \"kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.165793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.165737    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.165793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.165765    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.166435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.166412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.168268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.168242    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.189440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.189413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5j2f\" (UniqueName: \"kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.351739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.351707    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:43.660884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.660849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:43.696238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.696210    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"]
Oct 30 12:44:43.697043 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:43.697014    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd4ccac_8779_4218_943a_aa0d0b43a1f8.slice/crio-8253ee6226679024e55cfd9cdc315918b86fdb1a4a261209e14f804edc6533e6 WatchSource:0}: Error finding container 8253ee6226679024e55cfd9cdc315918b86fdb1a4a261209e14f804edc6533e6: Status 404 returned error can't find the container with id 8253ee6226679024e55cfd9cdc315918b86fdb1a4a261209e14f804edc6533e6
Oct 30 12:44:43.895752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.895705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5" event={"ID":"ccd4ccac-8779-4218-943a-aa0d0b43a1f8","Type":"ContainerStarted","Data":"8253ee6226679024e55cfd9cdc315918b86fdb1a4a261209e14f804edc6533e6"}
Oct 30 12:44:43.897959 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.897926    5060 generic.go:356] "Generic (PLEG): container finished" podID="1f04e82e-8a4d-499a-93e8-3ebea7e4120d" containerID="07959f34fa5dc6b34b038b10ecc80522e8f4767e0a2c64ef9ff6a93b7945ddd9" exitCode=0
Oct 30 12:44:43.898156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:43.898026    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts" event={"ID":"1f04e82e-8a4d-499a-93e8-3ebea7e4120d","Type":"ContainerDied","Data":"07959f34fa5dc6b34b038b10ecc80522e8f4767e0a2c64ef9ff6a93b7945ddd9"}
Oct 30 12:44:44.910657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:44.910625    5060 generic.go:356] "Generic (PLEG): container finished" podID="ccd4ccac-8779-4218-943a-aa0d0b43a1f8" containerID="ac97bbea3de340ab0b8cd625d62af527f55b9b6fa045aef6c28bfc055474cd86" exitCode=0
Oct 30 12:44:44.911107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:44.910711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5" event={"ID":"ccd4ccac-8779-4218-943a-aa0d0b43a1f8","Type":"ContainerDied","Data":"ac97bbea3de340ab0b8cd625d62af527f55b9b6fa045aef6c28bfc055474cd86"}
Oct 30 12:44:45.076073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.076047    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:45.090265 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.090234    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4\" (UniqueName: \"kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4\") pod \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") "
Oct 30 12:44:45.090400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.090309    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca\") pod \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") "
Oct 30 12:44:45.090447 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.090412    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca\") pod \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\" (UID: \"1f04e82e-8a4d-499a-93e8-3ebea7e4120d\") "
Oct 30 12:44:45.090927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.090901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca" pod "1f04e82e-8a4d-499a-93e8-3ebea7e4120d" (UID: "1f04e82e-8a4d-499a-93e8-3ebea7e4120d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:45.093320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.093290    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca" pod "1f04e82e-8a4d-499a-93e8-3ebea7e4120d" (UID: "1f04e82e-8a4d-499a-93e8-3ebea7e4120d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:45.093414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.093336    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4" pod "1f04e82e-8a4d-499a-93e8-3ebea7e4120d" (UID: "1f04e82e-8a4d-499a-93e8-3ebea7e4120d"). InnerVolumeSpecName "kube-api-access-ftnw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:45.132652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.132612    5060 patch_prober.go:29] interesting pod/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:44:45.132853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.132700    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:44:45.192103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.192059    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnw4\" (UniqueName: \"kubernetes.io/projected/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-kube-api-access-ftnw4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.192103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.192094    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.192103 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.192105    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e82e-8a4d-499a-93e8-3ebea7e4120d-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.216148 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.216044    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:45.441215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.441175    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:45.600537 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.597313    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:45.751283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.751246    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:45.851561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.851464    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"]
Oct 30 12:44:45.875477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.865800    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846099-mts2z"]
Oct 30 12:44:45.900225 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.900186    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:45.924196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.924161    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"
Oct 30 12:44:45.924196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.924182    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts" event={"ID":"1f04e82e-8a4d-499a-93e8-3ebea7e4120d","Type":"ContainerDied","Data":"1b03957cf815d9b9391d8d8fe15d39d4e02dd337dbdaa4a164edd545d1601fce"}
Oct 30 12:44:45.924793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:45.924271    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b03957cf815d9b9391d8d8fe15d39d4e02dd337dbdaa4a164edd545d1601fce"
Oct 30 12:44:46.091279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.091226    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:46.105629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.105554    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca\") pod \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") "
Oct 30 12:44:46.105790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.105626    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca\") pod \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") "
Oct 30 12:44:46.105790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.105663    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f\" (UniqueName: \"kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f\") pod \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\" (UID: \"ccd4ccac-8779-4218-943a-aa0d0b43a1f8\") "
Oct 30 12:44:46.105958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.105931    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca" pod "ccd4ccac-8779-4218-943a-aa0d0b43a1f8" (UID: "ccd4ccac-8779-4218-943a-aa0d0b43a1f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:46.106219 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.106198    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.108069 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.108046    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f" pod "ccd4ccac-8779-4218-943a-aa0d0b43a1f8" (UID: "ccd4ccac-8779-4218-943a-aa0d0b43a1f8"). InnerVolumeSpecName "kube-api-access-f5j2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:46.108173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.108128    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca" pod "ccd4ccac-8779-4218-943a-aa0d0b43a1f8" (UID: "ccd4ccac-8779-4218-943a-aa0d0b43a1f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:46.207610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.207573    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.207610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.207608    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5j2f\" (UniqueName: \"kubernetes.io/projected/ccd4ccac-8779-4218-943a-aa0d0b43a1f8-kube-api-access-f5j2f\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.345700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.345666    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc9a16c-d7d2-4e80-b637-95f2a199b707" path="/var/lib/kubelet/pods/dbc9a16c-d7d2-4e80-b637-95f2a199b707/volumes"
Oct 30 12:44:46.382539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.382431    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:46.695509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.695465    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:44:46.712629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712583    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.712824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712644    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.712824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712693    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.712824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712760    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.712824 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.713081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712843    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.713081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712898    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.713081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712993    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.713081 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713020    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy\") pod \"8abe7791-6018-475c-b11e-89f03f1206a3\" (UID: \"8abe7791-6018-475c-b11e-89f03f1206a3\") "
Oct 30 12:44:46.713310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.712895    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:46.713310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713211    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:46.713409 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713343    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:46.713810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713786    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.713908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713815    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abe7791-6018-475c-b11e-89f03f1206a3-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.713908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713831    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.713908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713896    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:46.714092 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.713913    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:46.715813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.715784    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "kube-api-access-kcrrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:46.716365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.716341    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:46.716509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.716415    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:44:46.720063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.720037    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy" pod "8abe7791-6018-475c-b11e-89f03f1206a3" (UID: "8abe7791-6018-475c-b11e-89f03f1206a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:46.814573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.814542    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.814573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.814570    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrrf\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-kube-api-access-kcrrf\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.814573 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.814581    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.814912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.814590    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abe7791-6018-475c-b11e-89f03f1206a3-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.814912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.814597    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abe7791-6018-475c-b11e-89f03f1206a3-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.814912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.814607    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abe7791-6018-475c-b11e-89f03f1206a3-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:46.954616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.954583    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"
Oct 30 12:44:46.955100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.954581    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5" event={"ID":"ccd4ccac-8779-4218-943a-aa0d0b43a1f8","Type":"ContainerDied","Data":"8253ee6226679024e55cfd9cdc315918b86fdb1a4a261209e14f804edc6533e6"}
Oct 30 12:44:46.955100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.954735    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8253ee6226679024e55cfd9cdc315918b86fdb1a4a261209e14f804edc6533e6"
Oct 30 12:44:46.957116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.957042    5060 generic.go:356] "Generic (PLEG): container finished" podID="8abe7791-6018-475c-b11e-89f03f1206a3" containerID="35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314" exitCode=0
Oct 30 12:44:46.957116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.957116    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"
Oct 30 12:44:46.957381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.957145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" event={"ID":"8abe7791-6018-475c-b11e-89f03f1206a3","Type":"ContainerDied","Data":"35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314"}
Oct 30 12:44:46.957381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.957191    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv" event={"ID":"8abe7791-6018-475c-b11e-89f03f1206a3","Type":"ContainerDied","Data":"317d3bae4ff1f821a6083a22bc8e259af23bdf0fabecc85fe1069f92d90964a3"}
Oct 30 12:44:46.957381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.957223    5060 scope.go:122] "RemoveContainer" containerID="35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314"
Oct 30 12:44:46.972039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.972014    5060 scope.go:122] "RemoveContainer" containerID="ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"
Oct 30 12:44:46.983209 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.983181    5060 scope.go:122] "RemoveContainer" containerID="35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314"
Oct 30 12:44:46.983539 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:46.983521    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314\": container with ID starting with 35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314 not found: ID does not exist" containerID="35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314"
Oct 30 12:44:46.983615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.983550    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314"} err="failed to get container status \"35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314\": rpc error: code = NotFound desc = could not find container \"35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314\": container with ID starting with 35f35f69e1d7db833d320109facda64b9605825cbf23be7fecc4b2e234b3e314 not found: ID does not exist"
Oct 30 12:44:46.983615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.983568    5060 scope.go:122] "RemoveContainer" containerID="ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"
Oct 30 12:44:46.983854 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:46.983836    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62\": container with ID starting with ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62 not found: ID does not exist" containerID="ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"
Oct 30 12:44:46.983910 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:46.983860    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62"} err="failed to get container status \"ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62\": rpc error: code = NotFound desc = could not find container \"ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62\": container with ID starting with ba14b9b442551b4927ad46a8955917d4b7da29029f0e91d9896e3329fa28ca62 not found: ID does not exist"
Oct 30 12:44:47.036499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:47.031578    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"]
Oct 30 12:44:47.036499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:47.035558    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bae155-openshift-default-6c7cd54585-qc5tv"]
Oct 30 12:44:47.973021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:47.972987    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:48.336137 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:48.336057    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" path="/var/lib/kubelet/pods/8abe7791-6018-475c-b11e-89f03f1206a3/volumes"
Oct 30 12:44:49.834093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:49.834060    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:49.875087 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:49.875049    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:44:51.655495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:51.655417    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:52.692267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:52.692229    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:52.842555 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:52.842512    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:53.338350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:53.338316    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:53.535949 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:53.535908    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:53.617284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:53.617185    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"]
Oct 30 12:44:53.617504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:53.617479    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy" containerID="cri-o://5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd" gracePeriod=30
Oct 30 12:44:53.694509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:53.694436    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:54.286387 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.286350    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"]
Oct 30 12:44:54.287138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287120    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy"
Oct 30 12:44:54.287194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287142    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy"
Oct 30 12:44:54.287194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287161    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy"
Oct 30 12:44:54.287194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287169    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy"
Oct 30 12:44:54.287194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287183    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ccd4ccac-8779-4218-943a-aa0d0b43a1f8" containerName="revoke-keys"
Oct 30 12:44:54.287194 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287189    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd4ccac-8779-4218-943a-aa0d0b43a1f8" containerName="revoke-keys"
Oct 30 12:44:54.287365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287244    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1f04e82e-8a4d-499a-93e8-3ebea7e4120d" containerName="revoke-keys"
Oct 30 12:44:54.287365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287252    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f04e82e-8a4d-499a-93e8-3ebea7e4120d" containerName="revoke-keys"
Oct 30 12:44:54.287432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287386    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy"
Oct 30 12:44:54.287432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287398    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd4ccac-8779-4218-943a-aa0d0b43a1f8" containerName="revoke-keys"
Oct 30 12:44:54.287432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.287417    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f04e82e-8a4d-499a-93e8-3ebea7e4120d" containerName="revoke-keys"
Oct 30 12:44:54.293176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.293153    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.308495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.308439    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:44:54.308654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.308569    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:44:54.311811 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.311785    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"]
Oct 30 12:44:54.373261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.373230    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:44:54.373394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.373277    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:44:54.442733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.442702    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtb48\" (UniqueName: \"kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.442937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.442798    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.442937 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.442866    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.544205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.544113    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.544205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.544173    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.544502 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.544268    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtb48\" (UniqueName: \"kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.544952 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.544931    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.546823 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.546795    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.567918 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.567892    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtb48\" (UniqueName: \"kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.611004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.610971    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:54.926585 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:44:54.926549    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b77256d_046d_4e4b_ae17_8a5e1142e574.slice/crio-8dcb1ab08bb6d742ada4637a3ce36d9599ac9636ccdb3f505acf8c212bfeb1b0 WatchSource:0}: Error finding container 8dcb1ab08bb6d742ada4637a3ce36d9599ac9636ccdb3f505acf8c212bfeb1b0: Status 404 returned error can't find the container with id 8dcb1ab08bb6d742ada4637a3ce36d9599ac9636ccdb3f505acf8c212bfeb1b0
Oct 30 12:44:54.927878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:54.927778    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"]
Oct 30 12:44:55.033739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:55.033707    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq" event={"ID":"6b77256d-046d-4e4b-ae17-8a5e1142e574","Type":"ContainerStarted","Data":"8dcb1ab08bb6d742ada4637a3ce36d9599ac9636ccdb3f505acf8c212bfeb1b0"}
Oct 30 12:44:55.467000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:55.466960    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:56.043336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:56.043249    5060 generic.go:356] "Generic (PLEG): container finished" podID="6b77256d-046d-4e4b-ae17-8a5e1142e574" containerID="34cb0c1337b139581ee26dadf0685dd1e7e5a3d3dfa9dfda9ea4bff48804519a" exitCode=0
Oct 30 12:44:56.043764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:56.043336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq" event={"ID":"6b77256d-046d-4e4b-ae17-8a5e1142e574","Type":"ContainerDied","Data":"34cb0c1337b139581ee26dadf0685dd1e7e5a3d3dfa9dfda9ea4bff48804519a"}
Oct 30 12:44:56.714992 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:56.714955    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:56.868354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:56.868313    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.208406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.208378    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:57.221298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.221260    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.380691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.380579    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca\") pod \"6b77256d-046d-4e4b-ae17-8a5e1142e574\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") "
Oct 30 12:44:57.380691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.380678    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca\") pod \"6b77256d-046d-4e4b-ae17-8a5e1142e574\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") "
Oct 30 12:44:57.380972 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.380844    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48\" (UniqueName: \"kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48\") pod \"6b77256d-046d-4e4b-ae17-8a5e1142e574\" (UID: \"6b77256d-046d-4e4b-ae17-8a5e1142e574\") "
Oct 30 12:44:57.381070 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.381036    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca" pod "6b77256d-046d-4e4b-ae17-8a5e1142e574" (UID: "6b77256d-046d-4e4b-ae17-8a5e1142e574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:57.381277 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.381258    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:57.382984 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.382955    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca" pod "6b77256d-046d-4e4b-ae17-8a5e1142e574" (UID: "6b77256d-046d-4e4b-ae17-8a5e1142e574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:57.383127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.383040    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48" pod "6b77256d-046d-4e4b-ae17-8a5e1142e574" (UID: "6b77256d-046d-4e4b-ae17-8a5e1142e574"). InnerVolumeSpecName "kube-api-access-jtb48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:57.452693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.452657    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.482558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.482524    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtb48\" (UniqueName: \"kubernetes.io/projected/6b77256d-046d-4e4b-ae17-8a5e1142e574-kube-api-access-jtb48\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:57.482558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:57.482558    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77256d-046d-4e4b-ae17-8a5e1142e574-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:58.060708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.060678    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"
Oct 30 12:44:58.060924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.060725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq" event={"ID":"6b77256d-046d-4e4b-ae17-8a5e1142e574","Type":"ContainerDied","Data":"8dcb1ab08bb6d742ada4637a3ce36d9599ac9636ccdb3f505acf8c212bfeb1b0"}
Oct 30 12:44:58.060924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.060785    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dcb1ab08bb6d742ada4637a3ce36d9599ac9636ccdb3f505acf8c212bfeb1b0"
Oct 30 12:44:58.213031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.213002    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:44:58.217230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.217207    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:44:58.891912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.891837    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:44:58.987955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:58.987914    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:59.001828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.001573    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002420    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002481    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002530    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002662    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002699    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002733    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002790    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.003698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.002817    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo\") pod \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\" (UID: \"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf\") "
Oct 30 12:44:59.005968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.005326    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:59.005968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.005709    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:59.005968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.005942    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:59.006529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.006498    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:59.006897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.006877    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:59.007854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.007821    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "kube-api-access-29dlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:59.011640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.011594    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:59.011640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.011606    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:59.011945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.011928    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo" pod "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" (UID: "90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:44:59.072818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.072776    5060 generic.go:356] "Generic (PLEG): container finished" podID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerID="5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd" exitCode=0
Oct 30 12:44:59.073038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.072839    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"
Oct 30 12:44:59.073038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.072866    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" event={"ID":"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf","Type":"ContainerDied","Data":"5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd"}
Oct 30 12:44:59.073038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.072938    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8" event={"ID":"90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf","Type":"ContainerDied","Data":"baf32d04f98ce5111c7a68fb5c68df0ec5c4b06122fb1ba0c52d6ead8ec0c82d"}
Oct 30 12:44:59.073038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.072975    5060 scope.go:122] "RemoveContainer" containerID="5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd"
Oct 30 12:44:59.086855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.086828    5060 scope.go:122] "RemoveContainer" containerID="9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"
Oct 30 12:44:59.098072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.098048    5060 scope.go:122] "RemoveContainer" containerID="5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd"
Oct 30 12:44:59.098494 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:59.098436    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd\": container with ID starting with 5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd not found: ID does not exist" containerID="5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd"
Oct 30 12:44:59.098608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.098507    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd"} err="failed to get container status \"5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd\": rpc error: code = NotFound desc = could not find container \"5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd\": container with ID starting with 5c2671cb5983d7db28aceea5b711b1b06a5c5d78bc9248c565dbe3445daf4dfd not found: ID does not exist"
Oct 30 12:44:59.098608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.098532    5060 scope.go:122] "RemoveContainer" containerID="9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"
Oct 30 12:44:59.098875 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:44:59.098834    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8\": container with ID starting with 9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8 not found: ID does not exist" containerID="9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"
Oct 30 12:44:59.098955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.098877    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8"} err="failed to get container status \"9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8\": rpc error: code = NotFound desc = could not find container \"9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8\": container with ID starting with 9528e64d0f321bcac3d2fbd46ff19751f728398c697ff9f9a57fa5d188a00ed8 not found: ID does not exist"
Oct 30 12:44:59.104711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104677    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104711 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104714    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104728    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104741    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104754    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104769    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29dlk\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-kube-api-access-29dlk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104782    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104797    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.104928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.104809    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:44:59.150571 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.149994    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"]
Oct 30 12:44:59.182171 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.182124    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-846099-openshift-default-66cbbc5485-cplw8"]
Oct 30 12:44:59.555654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.555619    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:59.648639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.648602    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"]
Oct 30 12:44:59.652803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.652781    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2c719a-ng5q5"]
Oct 30 12:44:59.693620 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:44:59.693576    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:00.160370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.160333    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"]
Oct 30 12:45:00.160991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.160970    5060 cpu_manager.go:414] "containerMap: removing container" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy"
Oct 30 12:45:00.160991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.160990    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy"
Oct 30 12:45:00.161169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.161009    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6b77256d-046d-4e4b-ae17-8a5e1142e574" containerName="revoke-keys"
Oct 30 12:45:00.161169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.161015    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b77256d-046d-4e4b-ae17-8a5e1142e574" containerName="revoke-keys"
Oct 30 12:45:00.161169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.161111    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b77256d-046d-4e4b-ae17-8a5e1142e574" containerName="revoke-keys"
Oct 30 12:45:00.161169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.161120    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy"
Oct 30 12:45:00.161169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.161128    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abe7791-6018-475c-b11e-89f03f1206a3" containerName="istio-proxy"
Oct 30 12:45:00.166144 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.166114    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"]
Oct 30 12:45:00.166309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.166225    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.166913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.166891    5060 cpu_manager.go:414] "containerMap: removing container" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy"
Oct 30 12:45:00.166913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.166912    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy"
Oct 30 12:45:00.167105 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.167046    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" containerName="istio-proxy"
Oct 30 12:45:00.170862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.170834    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"]
Oct 30 12:45:00.170987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.170944    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.175700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.175481    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"]
Oct 30 12:45:00.175790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.175757    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.180899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.180876    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.186794 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.186766    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:45:00.188889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.188858    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:45:00.189630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.189595    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:00.191187 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.190951    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:45:00.191870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.191844    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"]
Oct 30 12:45:00.198044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.196923    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"]
Oct 30 12:45:00.214178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.212885    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"]
Oct 30 12:45:00.258477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.257541    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"]
Oct 30 12:45:00.269727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.269687    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.269939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.269754    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.269939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.269774    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.269939 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.269876    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.270136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.269946    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzn4\" (UniqueName: \"kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.270136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.270062    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwl4\" (UniqueName: \"kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.270136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.270117    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hhn\" (UniqueName: \"kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.270294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.270149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrhd\" (UniqueName: \"kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.270294 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.270197    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.272861 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.272830    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"]
Oct 30 12:45:00.277785 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.277691    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.286776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.283167    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"]
Oct 30 12:45:00.297575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.297550    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"]
Oct 30 12:45:00.305630 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.305591    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"]
Oct 30 12:45:00.334340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.334299    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf" path="/var/lib/kubelet/pods/90cf13b3-fb7d-474b-b5f2-4a8c4d34bddf/volumes"
Oct 30 12:45:00.334919 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.334896    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd4ccac-8779-4218-943a-aa0d0b43a1f8" path="/var/lib/kubelet/pods/ccd4ccac-8779-4218-943a-aa0d0b43a1f8/volumes"
Oct 30 12:45:00.371779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.371734    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.371991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.371789    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw24c\" (UniqueName: \"kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.371991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.371847    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.371991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.371885    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzn4\" (UniqueName: \"kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.371991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.371913    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwl4\" (UniqueName: \"kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.371991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.371943    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hhn\" (UniqueName: \"kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.372298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372061    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrhd\" (UniqueName: \"kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.372298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372114    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.372298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372188    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.372298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372242    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5qs\" (UniqueName: \"kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.372298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372283    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.372641 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372363    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.372783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.372956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372935    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.373030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.372971    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.373074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.373059    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.397931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.397896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwl4\" (UniqueName: \"kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.398196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.397900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrhd\" (UniqueName: \"kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.398331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.398309    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzn4\" (UniqueName: \"kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.398623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.398606    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hhn\" (UniqueName: \"kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.473765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.473721    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw24c\" (UniqueName: \"kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.474017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.473913    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5qs\" (UniqueName: \"kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.474017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.473955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.474017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.474007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.474669 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.474643    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.474752 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.474649    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.488768 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.488740    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:00.498279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.498243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw24c\" (UniqueName: \"kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.498426 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.498295    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5qs\" (UniqueName: \"kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.508732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.508703    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:00.579420 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.579380    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:00.595844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.595811    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:00.622189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.612770    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:00.622189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.619663    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:00.815648 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:00.815611    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe88a80_a679_428d_a502_f1e1003dc619.slice/crio-aef145245063bfc5687255478706a0dbb9893c690ab231574583a97b862d339d WatchSource:0}: Error finding container aef145245063bfc5687255478706a0dbb9893c690ab231574583a97b862d339d: Status 404 returned error can't find the container with id aef145245063bfc5687255478706a0dbb9893c690ab231574583a97b862d339d
Oct 30 12:45:00.819342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.818976    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"]
Oct 30 12:45:00.840548 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:00.840516    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aedbee_5907_4b9e_b548_19f65dd1314d.slice/crio-2043daeeecacb21613b87eaba3449fa8cdd32cab71de3b95ec2d2621dfa5a68c WatchSource:0}: Error finding container 2043daeeecacb21613b87eaba3449fa8cdd32cab71de3b95ec2d2621dfa5a68c: Status 404 returned error can't find the container with id 2043daeeecacb21613b87eaba3449fa8cdd32cab71de3b95ec2d2621dfa5a68c
Oct 30 12:45:00.842332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.842273    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"]
Oct 30 12:45:00.929483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.929426    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"]
Oct 30 12:45:00.961653 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.961620    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"]
Oct 30 12:45:00.979423 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:00.979386    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89785f16_1a3f_48d0_b3dc_21bb8f40d175.slice/crio-25f6050d5e0d67ab832d5ae56fe68375189844bc8e962db21fb6aacdde1064c2 WatchSource:0}: Error finding container 25f6050d5e0d67ab832d5ae56fe68375189844bc8e962db21fb6aacdde1064c2: Status 404 returned error can't find the container with id 25f6050d5e0d67ab832d5ae56fe68375189844bc8e962db21fb6aacdde1064c2
Oct 30 12:45:00.980857 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:00.980831    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1040b2a_4038_4d24_9bb0_856aecc50dda.slice/crio-187ebcb4dc4b0c34662d62ce1a7fc44da266e6f437bef2b1656ab09853f1165a WatchSource:0}: Error finding container 187ebcb4dc4b0c34662d62ce1a7fc44da266e6f437bef2b1656ab09853f1165a: Status 404 returned error can't find the container with id 187ebcb4dc4b0c34662d62ce1a7fc44da266e6f437bef2b1656ab09853f1165a
Oct 30 12:45:00.993150 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:00.993127    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"]
Oct 30 12:45:00.995314 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:00.995280    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7cb7f0_4da4_4b96_84c7_6e3a0fcf236a.slice/crio-5dc2d6b4f5d8f60ea0e6dbbcb1c8d0a5fa139fe43a36650e6637e6ca2abd0ed4 WatchSource:0}: Error finding container 5dc2d6b4f5d8f60ea0e6dbbcb1c8d0a5fa139fe43a36650e6637e6ca2abd0ed4: Status 404 returned error can't find the container with id 5dc2d6b4f5d8f60ea0e6dbbcb1c8d0a5fa139fe43a36650e6637e6ca2abd0ed4
Oct 30 12:45:01.006619 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:01.006580    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1263bb2_a207_4313_bdfa_1ec1a96b1b25.slice/crio-abf24dc3fe4632d025c59c82edd0d9689b1ce4cc73e84eb9d67d91a265d07ed4 WatchSource:0}: Error finding container abf24dc3fe4632d025c59c82edd0d9689b1ce4cc73e84eb9d67d91a265d07ed4: Status 404 returned error can't find the container with id abf24dc3fe4632d025c59c82edd0d9689b1ce4cc73e84eb9d67d91a265d07ed4
Oct 30 12:45:01.007329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.007300    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"]
Oct 30 12:45:01.102639 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.102500    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" event={"ID":"7fe88a80-a679-428d-a502-f1e1003dc619","Type":"ContainerStarted","Data":"aef145245063bfc5687255478706a0dbb9893c690ab231574583a97b862d339d"}
Oct 30 12:45:01.104312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.104274    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" event={"ID":"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a","Type":"ContainerStarted","Data":"5dc2d6b4f5d8f60ea0e6dbbcb1c8d0a5fa139fe43a36650e6637e6ca2abd0ed4"}
Oct 30 12:45:01.105743 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.105711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerStarted","Data":"abf24dc3fe4632d025c59c82edd0d9689b1ce4cc73e84eb9d67d91a265d07ed4"}
Oct 30 12:45:01.107094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.107067    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerStarted","Data":"25f6050d5e0d67ab832d5ae56fe68375189844bc8e962db21fb6aacdde1064c2"}
Oct 30 12:45:01.108498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.108477    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerStarted","Data":"2043daeeecacb21613b87eaba3449fa8cdd32cab71de3b95ec2d2621dfa5a68c"}
Oct 30 12:45:01.109776 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.109756    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerStarted","Data":"187ebcb4dc4b0c34662d62ce1a7fc44da266e6f437bef2b1656ab09853f1165a"}
Oct 30 12:45:01.755321 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:01.755282    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:02.121758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.121726    5060 generic.go:356] "Generic (PLEG): container finished" podID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerID="8ea6c7aa2ebed48ad5757bb8cb4b9d4b5e6885715a80c0abeec123b153df5391" exitCode=60
Oct 30 12:45:02.121950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.121804    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerDied","Data":"8ea6c7aa2ebed48ad5757bb8cb4b9d4b5e6885715a80c0abeec123b153df5391"}
Oct 30 12:45:02.122169 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.122153    5060 scope.go:122] "RemoveContainer" containerID="8ea6c7aa2ebed48ad5757bb8cb4b9d4b5e6885715a80c0abeec123b153df5391"
Oct 30 12:45:02.123960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.123937    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerStarted","Data":"425d51d49623a930a483a207ad7d65a81738b19caa34a1fc0087ff82b551dd23"}
Oct 30 12:45:02.125867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.125727    5060 generic.go:356] "Generic (PLEG): container finished" podID="7fe88a80-a679-428d-a502-f1e1003dc619" containerID="65ad0936d8bf95c4abde1738e56713f77126449b4172c9050dbd9144a252874c" exitCode=60
Oct 30 12:45:02.125867 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.125816    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" event={"ID":"7fe88a80-a679-428d-a502-f1e1003dc619","Type":"ContainerDied","Data":"65ad0936d8bf95c4abde1738e56713f77126449b4172c9050dbd9144a252874c"}
Oct 30 12:45:02.126072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.126054    5060 scope.go:122] "RemoveContainer" containerID="65ad0936d8bf95c4abde1738e56713f77126449b4172c9050dbd9144a252874c"
Oct 30 12:45:02.127786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.127728    5060 generic.go:356] "Generic (PLEG): container finished" podID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerID="66c7fde06a3ed255cf5f391377fd98639c92ba2fbf6b5d023a18794298a32c03" exitCode=60
Oct 30 12:45:02.127889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.127797    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" event={"ID":"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a","Type":"ContainerDied","Data":"66c7fde06a3ed255cf5f391377fd98639c92ba2fbf6b5d023a18794298a32c03"}
Oct 30 12:45:02.128037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.128022    5060 scope.go:122] "RemoveContainer" containerID="66c7fde06a3ed255cf5f391377fd98639c92ba2fbf6b5d023a18794298a32c03"
Oct 30 12:45:02.129731 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.129704    5060 generic.go:356] "Generic (PLEG): container finished" podID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerID="60d80556c3eaa911048258fbfd647d2248e34ef09b660fed7cfa22a14edcbaa8" exitCode=60
Oct 30 12:45:02.129862 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.129789    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerDied","Data":"60d80556c3eaa911048258fbfd647d2248e34ef09b660fed7cfa22a14edcbaa8"}
Oct 30 12:45:02.129960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.129943    5060 scope.go:122] "RemoveContainer" containerID="60d80556c3eaa911048258fbfd647d2248e34ef09b660fed7cfa22a14edcbaa8"
Oct 30 12:45:02.131897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.131859    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerStarted","Data":"5b503c5b788c19065896e47ae4dd25ba0fa0e0ce96e1d5e9869fdf739fd7cf78"}
Oct 30 12:45:02.293269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.293217    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" podStartSLOduration=2.293204276 podStartE2EDuration="2.293204276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.289416416 +0000 UTC m=+4506.084300907" watchObservedRunningTime="2026-10-30 12:45:02.293204276 +0000 UTC m=+4506.088088769"
Oct 30 12:45:02.374095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:02.374045    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" podStartSLOduration=2.374030037 podStartE2EDuration="2.374030037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.372753793 +0000 UTC m=+4506.167638287" watchObservedRunningTime="2026-10-30 12:45:02.374030037 +0000 UTC m=+4506.168914531"
Oct 30 12:45:03.143504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.143377    5060 generic.go:356] "Generic (PLEG): container finished" podID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerID="425d51d49623a930a483a207ad7d65a81738b19caa34a1fc0087ff82b551dd23" exitCode=60
Oct 30 12:45:03.143504 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.143480    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerDied","Data":"425d51d49623a930a483a207ad7d65a81738b19caa34a1fc0087ff82b551dd23"}
Oct 30 12:45:03.144094 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.143820    5060 scope.go:122] "RemoveContainer" containerID="425d51d49623a930a483a207ad7d65a81738b19caa34a1fc0087ff82b551dd23"
Oct 30 12:45:03.145784 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.145758    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" event={"ID":"7fe88a80-a679-428d-a502-f1e1003dc619","Type":"ContainerStarted","Data":"9f66785597952312a160f528fe45d3537869cc6ca4db2febcf1483972beaeb4f"}
Oct 30 12:45:03.148090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.148068    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" event={"ID":"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a","Type":"ContainerStarted","Data":"73fd9bc9f19cbae0f50456e2c98fdd73af707d5a843bb0d486a86ecb7e72ebd6"}
Oct 30 12:45:03.150052 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.150017    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerStarted","Data":"4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645"}
Oct 30 12:45:03.151730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.151707    5060 generic.go:356] "Generic (PLEG): container finished" podID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerID="5b503c5b788c19065896e47ae4dd25ba0fa0e0ce96e1d5e9869fdf739fd7cf78" exitCode=60
Oct 30 12:45:03.151836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.151756    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerDied","Data":"5b503c5b788c19065896e47ae4dd25ba0fa0e0ce96e1d5e9869fdf739fd7cf78"}
Oct 30 12:45:03.151986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.151969    5060 scope.go:122] "RemoveContainer" containerID="5b503c5b788c19065896e47ae4dd25ba0fa0e0ce96e1d5e9869fdf739fd7cf78"
Oct 30 12:45:03.153799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.153782    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerStarted","Data":"2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c"}
Oct 30 12:45:03.231175 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.231119    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" podStartSLOduration=3.231104032 podStartE2EDuration="3.231104032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.226721604 +0000 UTC m=+4507.021606098" watchObservedRunningTime="2026-10-30 12:45:03.231104032 +0000 UTC m=+4507.025988528"
Oct 30 12:45:03.313278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.313223    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" podStartSLOduration=3.3132079709999998 podStartE2EDuration="3.313207971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.30971953 +0000 UTC m=+4507.104604025" watchObservedRunningTime="2026-10-30 12:45:03.313207971 +0000 UTC m=+4507.108092468"
Oct 30 12:45:03.353306 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.353243    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" podStartSLOduration=3.353231468 podStartE2EDuration="3.353231468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.350472734 +0000 UTC m=+4507.145357221" watchObservedRunningTime="2026-10-30 12:45:03.353231468 +0000 UTC m=+4507.148115963"
Oct 30 12:45:03.392720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.392665    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" podStartSLOduration=3.392650599 podStartE2EDuration="3.392650599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.390860382 +0000 UTC m=+4507.185744876" watchObservedRunningTime="2026-10-30 12:45:03.392650599 +0000 UTC m=+4507.187535099"
Oct 30 12:45:03.639664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.639626    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:03.960789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:03.960743    5060 scope.go:122] "RemoveContainer" containerID="35b1d50f2a2d388d4d17885bf5b9c7e39f00e17748c4787383bd06baa03d8724"
Oct 30 12:45:04.163042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.163002    5060 generic.go:356] "Generic (PLEG): container finished" podID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerID="4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645" exitCode=60
Oct 30 12:45:04.163535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.163084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerDied","Data":"4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645"}
Oct 30 12:45:04.163535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.163173    5060 scope.go:122] "RemoveContainer" containerID="60d80556c3eaa911048258fbfd647d2248e34ef09b660fed7cfa22a14edcbaa8"
Oct 30 12:45:04.163658 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.163533    5060 scope.go:122] "RemoveContainer" containerID="4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645"
Oct 30 12:45:04.163878 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:04.163854    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64_odh-ai-gateway-infra(d1263bb2-a207-4313-bdfa-1ec1a96b1b25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25"
Oct 30 12:45:04.165406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.165383    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerStarted","Data":"c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5"}
Oct 30 12:45:04.167216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.167185    5060 generic.go:356] "Generic (PLEG): container finished" podID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerID="2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c" exitCode=60
Oct 30 12:45:04.167320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.167264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerDied","Data":"2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c"}
Oct 30 12:45:04.167565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.167533    5060 scope.go:122] "RemoveContainer" containerID="2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c"
Oct 30 12:45:04.167780 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:04.167761    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc_odh-ai-gateway-infra(c6aedbee-5907-4b9e-b548-19f65dd1314d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d"
Oct 30 12:45:04.169345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.169318    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerStarted","Data":"42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66"}
Oct 30 12:45:04.171238 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.171217    5060 generic.go:356] "Generic (PLEG): container finished" podID="7fe88a80-a679-428d-a502-f1e1003dc619" containerID="9f66785597952312a160f528fe45d3537869cc6ca4db2febcf1483972beaeb4f" exitCode=60
Oct 30 12:45:04.171342 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.171297    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" event={"ID":"7fe88a80-a679-428d-a502-f1e1003dc619","Type":"ContainerDied","Data":"9f66785597952312a160f528fe45d3537869cc6ca4db2febcf1483972beaeb4f"}
Oct 30 12:45:04.171500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.171480    5060 scope.go:122] "RemoveContainer" containerID="9f66785597952312a160f528fe45d3537869cc6ca4db2febcf1483972beaeb4f"
Oct 30 12:45:04.171679 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:04.171663    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm_odh-ai-gateway-infra(7fe88a80-a679-428d-a502-f1e1003dc619)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" podUID="7fe88a80-a679-428d-a502-f1e1003dc619"
Oct 30 12:45:04.173499 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.173474    5060 generic.go:356] "Generic (PLEG): container finished" podID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerID="73fd9bc9f19cbae0f50456e2c98fdd73af707d5a843bb0d486a86ecb7e72ebd6" exitCode=60
Oct 30 12:45:04.173616 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.173525    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" event={"ID":"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a","Type":"ContainerDied","Data":"73fd9bc9f19cbae0f50456e2c98fdd73af707d5a843bb0d486a86ecb7e72ebd6"}
Oct 30 12:45:04.173724 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.173709    5060 scope.go:122] "RemoveContainer" containerID="73fd9bc9f19cbae0f50456e2c98fdd73af707d5a843bb0d486a86ecb7e72ebd6"
Oct 30 12:45:04.173918 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:04.173899    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72_odh-ai-gateway-infra(ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a"
Oct 30 12:45:04.182324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.182306    5060 scope.go:122] "RemoveContainer" containerID="8ea6c7aa2ebed48ad5757bb8cb4b9d4b5e6885715a80c0abeec123b153df5391"
Oct 30 12:45:04.244031 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.244010    5060 scope.go:122] "RemoveContainer" containerID="65ad0936d8bf95c4abde1738e56713f77126449b4172c9050dbd9144a252874c"
Oct 30 12:45:04.259224 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.259201    5060 scope.go:122] "RemoveContainer" containerID="66c7fde06a3ed255cf5f391377fd98639c92ba2fbf6b5d023a18794298a32c03"
Oct 30 12:45:04.772217 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.772179    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"]
Oct 30 12:45:04.804221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.804185    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"]
Oct 30 12:45:04.942060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:04.942029    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:05.087635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.087539    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:05.185393 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.185354    5060 generic.go:356] "Generic (PLEG): container finished" podID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerID="c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5" exitCode=60
Oct 30 12:45:05.185945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.185442    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerDied","Data":"c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5"}
Oct 30 12:45:05.185945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.185525    5060 scope.go:122] "RemoveContainer" containerID="5b503c5b788c19065896e47ae4dd25ba0fa0e0ce96e1d5e9869fdf739fd7cf78"
Oct 30 12:45:05.185945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.185819    5060 scope.go:122] "RemoveContainer" containerID="c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5"
Oct 30 12:45:05.186261 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:05.186165    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d_odh-ai-gateway-infra(89785f16-1a3f-48d0-b3dc-21bb8f40d175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175"
Oct 30 12:45:05.187943 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.187926    5060 scope.go:122] "RemoveContainer" containerID="2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c"
Oct 30 12:45:05.188219 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:05.188190    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc_odh-ai-gateway-infra(c6aedbee-5907-4b9e-b548-19f65dd1314d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d"
Oct 30 12:45:05.190088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.189847    5060 generic.go:356] "Generic (PLEG): container finished" podID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerID="42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66" exitCode=60
Oct 30 12:45:05.190088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.189923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerDied","Data":"42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66"}
Oct 30 12:45:05.190272 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.190162    5060 scope.go:122] "RemoveContainer" containerID="42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66"
Oct 30 12:45:05.190503 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:05.190442    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5_odh-ai-gateway-infra(d1040b2a-4038-4d24-9bb0-856aecc50dda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda"
Oct 30 12:45:05.192349 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.192330    5060 scope.go:122] "RemoveContainer" containerID="9f66785597952312a160f528fe45d3537869cc6ca4db2febcf1483972beaeb4f"
Oct 30 12:45:05.192668 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:05.192642    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm_odh-ai-gateway-infra(7fe88a80-a679-428d-a502-f1e1003dc619)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" podUID="7fe88a80-a679-428d-a502-f1e1003dc619"
Oct 30 12:45:05.194239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.194218    5060 scope.go:122] "RemoveContainer" containerID="73fd9bc9f19cbae0f50456e2c98fdd73af707d5a843bb0d486a86ecb7e72ebd6"
Oct 30 12:45:05.194508 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:05.194446    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72_odh-ai-gateway-infra(ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a"
Oct 30 12:45:05.196274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.196258    5060 scope.go:122] "RemoveContainer" containerID="4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645"
Oct 30 12:45:05.196488 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:05.196436    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64_odh-ai-gateway-infra(d1263bb2-a207-4313-bdfa-1ec1a96b1b25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25"
Oct 30 12:45:05.204250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.204227    5060 scope.go:122] "RemoveContainer" containerID="425d51d49623a930a483a207ad7d65a81738b19caa34a1fc0087ff82b551dd23"
Oct 30 12:45:05.419043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:05.418945    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:06.208805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:06.208775    5060 scope.go:122] "RemoveContainer" containerID="42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66"
Oct 30 12:45:06.209240 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:06.209090    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5_odh-ai-gateway-infra(d1040b2a-4038-4d24-9bb0-856aecc50dda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda"
Oct 30 12:45:06.210765 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:06.210743    5060 scope.go:122] "RemoveContainer" containerID="c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5"
Oct 30 12:45:06.210987 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:06.210969    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d_odh-ai-gateway-infra(89785f16-1a3f-48d0-b3dc-21bb8f40d175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175"
Oct 30 12:45:07.332145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:07.332076    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.561177 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:07.561131    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.738989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:07.738944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.897985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:07.897944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:08.987029 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:08.986963    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"]
Oct 30 12:45:09.002705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.002671    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.200477 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.200210    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:09.247536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.243793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72" event={"ID":"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a","Type":"ContainerDied","Data":"5dc2d6b4f5d8f60ea0e6dbbcb1c8d0a5fa139fe43a36650e6637e6ca2abd0ed4"}
Oct 30 12:45:09.247536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.243865    5060 scope.go:122] "RemoveContainer" containerID="73fd9bc9f19cbae0f50456e2c98fdd73af707d5a843bb0d486a86ecb7e72ebd6"
Oct 30 12:45:09.247536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.243803    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"
Oct 30 12:45:09.289095 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.289065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c\" (UniqueName: \"kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c\") pod \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") "
Oct 30 12:45:09.289250 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.289185    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca\") pod \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\" (UID: \"ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a\") "
Oct 30 12:45:09.289635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.289611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca" pod "ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" (UID: "ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:09.291436 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.291414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c" pod "ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" (UID: "ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a"). InnerVolumeSpecName "kube-api-access-fw24c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:09.391073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.391039    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw24c\" (UniqueName: \"kubernetes.io/projected/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-kube-api-access-fw24c\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:09.391073 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.391074    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:09.581204 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.581166    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.591078 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.591044    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"]
Oct 30 12:45:09.603374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.603342    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-87f2e6-29889405-w5m72"]
Oct 30 12:45:09.702279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.702244    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"]
Oct 30 12:45:09.725482 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.723109    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-87f2e6-msbts"]
Oct 30 12:45:09.730665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.730628    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.773506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.773472    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerName="payload-processing" containerID="cri-o://a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8" gracePeriod=25
Oct 30 12:45:09.804568 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:09.804488    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" podUID="9fbf6674-283c-43de-810e-94f436247a5a" containerName="payload-pre-processing" containerID="cri-o://916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8" gracePeriod=25
Oct 30 12:45:10.234983 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:10.234949    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:10.335432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:10.335387    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f04e82e-8a4d-499a-93e8-3ebea7e4120d" path="/var/lib/kubelet/pods/1f04e82e-8a4d-499a-93e8-3ebea7e4120d/volumes"
Oct 30 12:45:10.335915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:10.335893    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" path="/var/lib/kubelet/pods/ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a/volumes"
Oct 30 12:45:10.663817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:10.663733    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.063553 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.063511    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" podUID="9fbf6674-283c-43de-810e-94f436247a5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:11.063755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.063573    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:11.236288 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.236252    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"]
Oct 30 12:45:11.262831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.262799    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.467093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.467060    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:11.521965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.521932    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd\" (UniqueName: \"kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd\") pod \"7fe88a80-a679-428d-a502-f1e1003dc619\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") "
Oct 30 12:45:11.522635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.522610    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca\") pod \"7fe88a80-a679-428d-a502-f1e1003dc619\" (UID: \"7fe88a80-a679-428d-a502-f1e1003dc619\") "
Oct 30 12:45:11.523066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.523037    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca" pod "7fe88a80-a679-428d-a502-f1e1003dc619" (UID: "7fe88a80-a679-428d-a502-f1e1003dc619"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:11.525494 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.523437    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe88a80-a679-428d-a502-f1e1003dc619-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:11.525976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.525948    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd" pod "7fe88a80-a679-428d-a502-f1e1003dc619" (UID: "7fe88a80-a679-428d-a502-f1e1003dc619"). InnerVolumeSpecName "kube-api-access-xgrhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:11.640398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.639305    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrhd\" (UniqueName: \"kubernetes.io/projected/7fe88a80-a679-428d-a502-f1e1003dc619-kube-api-access-xgrhd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:11.823558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.823508    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.977501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.972269    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"]
Oct 30 12:45:11.979135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.979082    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f1b14f-9vpvq"]
Oct 30 12:45:11.999128 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:11.999090    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:12.317227 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.317111    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm" event={"ID":"7fe88a80-a679-428d-a502-f1e1003dc619","Type":"ContainerDied","Data":"aef145245063bfc5687255478706a0dbb9893c690ab231574583a97b862d339d"}
Oct 30 12:45:12.317885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.317855    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"
Oct 30 12:45:12.318045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.317874    5060 scope.go:122] "RemoveContainer" containerID="9f66785597952312a160f528fe45d3537869cc6ca4db2febcf1483972beaeb4f"
Oct 30 12:45:12.344974 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.344939    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b77256d-046d-4e4b-ae17-8a5e1142e574" path="/var/lib/kubelet/pods/6b77256d-046d-4e4b-ae17-8a5e1142e574/volumes"
Oct 30 12:45:12.377535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.377501    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"]
Oct 30 12:45:12.384299 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.384259    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f1b14f-29889405-xx9nm"]
Oct 30 12:45:12.480106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:12.480063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.034047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:13.034007    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"]
Oct 30 12:45:13.044004 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:13.043972    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4349d059-42krn"]
Oct 30 12:45:13.834166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:13.834109    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.334578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:14.334538    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fa000d-bf58-4d5f-9e83-4f0b5a056a0c" path="/var/lib/kubelet/pods/67fa000d-bf58-4d5f-9e83-4f0b5a056a0c/volumes"
Oct 30 12:45:14.335047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:14.335026    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" path="/var/lib/kubelet/pods/7fe88a80-a679-428d-a502-f1e1003dc619/volumes"
Oct 30 12:45:15.329586 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:15.329555    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:15.478104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:15.478068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:15.639956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:15.639865    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:17.297996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.297958    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:17.327508 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.327446    5060 scope.go:122] "RemoveContainer" containerID="2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c"
Oct 30 12:45:17.327666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.327653    5060 scope.go:122] "RemoveContainer" containerID="42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66"
Oct 30 12:45:17.485199 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.485107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:17.575491 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.575392    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"]
Oct 30 12:45:17.575685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.575644    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" containerID="cri-o://5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce" gracePeriod=30
Oct 30 12:45:17.660603 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:17.660555    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:18.327365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:18.327329    5060 scope.go:122] "RemoveContainer" containerID="c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5"
Oct 30 12:45:18.327912 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:18.327429    5060 scope.go:122] "RemoveContainer" containerID="4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645"
Oct 30 12:45:18.369831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:18.369794    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerStarted","Data":"1cbfbab593eb553cdec24ddf3fbf6ae5ecfe058d80d6e9cb22b90bd6648b2fb2"}
Oct 30 12:45:18.371730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:18.371708    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerStarted","Data":"de0b5ae46bb8ebf29a831bfcd5e2f8d786132d66a755e323d93cb855831e67a7"}
Oct 30 12:45:18.578509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:18.578143    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:45:18.578509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:18.578206    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:45:19.039873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.039840    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"]
Oct 30 12:45:19.086305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.086271    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"]
Oct 30 12:45:19.234664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.234627    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:19.383522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.383395    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerStarted","Data":"5ffd019bc91fe739936e7ba0005ee0a4d99bbff6300f4721e313c4fe93159183"}
Oct 30 12:45:19.386424 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.386397    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerStarted","Data":"a55d03a0ddc1a7d7f188385a2c51be444cb3cfb2c51c1d3cf7635640fd8bbf1c"}
Oct 30 12:45:19.393033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.393002    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:19.393980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.393958    5060 generic.go:356] "Generic (PLEG): container finished" podID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerID="1cbfbab593eb553cdec24ddf3fbf6ae5ecfe058d80d6e9cb22b90bd6648b2fb2" exitCode=60
Oct 30 12:45:19.394127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.394033    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerDied","Data":"1cbfbab593eb553cdec24ddf3fbf6ae5ecfe058d80d6e9cb22b90bd6648b2fb2"}
Oct 30 12:45:19.394127 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.394099    5060 scope.go:122] "RemoveContainer" containerID="2f9a73fa56fcb8fac6ab219d91bceb862e51490a1ec8c56ce3a4903983f9fd2c"
Oct 30 12:45:19.394483 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.394435    5060 scope.go:122] "RemoveContainer" containerID="1cbfbab593eb553cdec24ddf3fbf6ae5ecfe058d80d6e9cb22b90bd6648b2fb2"
Oct 30 12:45:19.394769 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:19.394747    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc_odh-ai-gateway-infra(c6aedbee-5907-4b9e-b548-19f65dd1314d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d"
Oct 30 12:45:19.396584 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.396552    5060 generic.go:356] "Generic (PLEG): container finished" podID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerID="de0b5ae46bb8ebf29a831bfcd5e2f8d786132d66a755e323d93cb855831e67a7" exitCode=60
Oct 30 12:45:19.396691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.396668    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerDied","Data":"de0b5ae46bb8ebf29a831bfcd5e2f8d786132d66a755e323d93cb855831e67a7"}
Oct 30 12:45:19.397038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.397022    5060 scope.go:122] "RemoveContainer" containerID="de0b5ae46bb8ebf29a831bfcd5e2f8d786132d66a755e323d93cb855831e67a7"
Oct 30 12:45:19.397265 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:19.397247    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5_odh-ai-gateway-infra(d1040b2a-4038-4d24-9bb0-856aecc50dda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda"
Oct 30 12:45:19.439750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.439721    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"]
Oct 30 12:45:19.443221 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.443200    5060 scope.go:122] "RemoveContainer" containerID="42eaad277a8a911f8390c9a9efb4806175fdf286cf78db0dc95bafe269b15b66"
Oct 30 12:45:19.505193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.505160    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"]
Oct 30 12:45:19.642813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:19.642733    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:19.864776 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:19.864734    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:45:20.411145 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.411109    5060 generic.go:356] "Generic (PLEG): container finished" podID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerID="5ffd019bc91fe739936e7ba0005ee0a4d99bbff6300f4721e313c4fe93159183" exitCode=60
Oct 30 12:45:20.411615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.411188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerDied","Data":"5ffd019bc91fe739936e7ba0005ee0a4d99bbff6300f4721e313c4fe93159183"}
Oct 30 12:45:20.411615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.411267    5060 scope.go:122] "RemoveContainer" containerID="4aec02fafcae78ac1b44074e1af26038fc0e16ccc9dbb227a5cf2de2e6a77645"
Oct 30 12:45:20.411764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.411674    5060 scope.go:122] "RemoveContainer" containerID="5ffd019bc91fe739936e7ba0005ee0a4d99bbff6300f4721e313c4fe93159183"
Oct 30 12:45:20.412039 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:20.412013    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64_odh-ai-gateway-infra(d1263bb2-a207-4313-bdfa-1ec1a96b1b25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25"
Oct 30 12:45:20.413530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.413510    5060 generic.go:356] "Generic (PLEG): container finished" podID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerID="a55d03a0ddc1a7d7f188385a2c51be444cb3cfb2c51c1d3cf7635640fd8bbf1c" exitCode=60
Oct 30 12:45:20.413640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.413569    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerDied","Data":"a55d03a0ddc1a7d7f188385a2c51be444cb3cfb2c51c1d3cf7635640fd8bbf1c"}
Oct 30 12:45:20.413945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.413929    5060 scope.go:122] "RemoveContainer" containerID="a55d03a0ddc1a7d7f188385a2c51be444cb3cfb2c51c1d3cf7635640fd8bbf1c"
Oct 30 12:45:20.414257 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:20.414209    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d_odh-ai-gateway-infra(89785f16-1a3f-48d0-b3dc-21bb8f40d175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175"
Oct 30 12:45:20.428189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.428171    5060 scope.go:122] "RemoveContainer" containerID="c3b2eb8255a70a530396a2e1aaccd15039a4c478b06f08e22dd6806fb2687fc5"
Oct 30 12:45:20.444705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.444670    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"]
Oct 30 12:45:20.530213 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.528842    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"]
Oct 30 12:45:20.679766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.679742    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:20.682112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.682091    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:20.775605 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.775573    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4\" (UniqueName: \"kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4\") pod \"d1040b2a-4038-4d24-9bb0-856aecc50dda\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") "
Oct 30 12:45:20.775806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.775628    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4\" (UniqueName: \"kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4\") pod \"c6aedbee-5907-4b9e-b548-19f65dd1314d\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") "
Oct 30 12:45:20.775806 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.775787    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca\") pod \"c6aedbee-5907-4b9e-b548-19f65dd1314d\" (UID: \"c6aedbee-5907-4b9e-b548-19f65dd1314d\") "
Oct 30 12:45:20.776017 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.775834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca\") pod \"d1040b2a-4038-4d24-9bb0-856aecc50dda\" (UID: \"d1040b2a-4038-4d24-9bb0-856aecc50dda\") "
Oct 30 12:45:20.776186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.776160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca" pod "c6aedbee-5907-4b9e-b548-19f65dd1314d" (UID: "c6aedbee-5907-4b9e-b548-19f65dd1314d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:20.776259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.776234    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca" pod "d1040b2a-4038-4d24-9bb0-856aecc50dda" (UID: "d1040b2a-4038-4d24-9bb0-856aecc50dda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:20.777928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.777905    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4" pod "d1040b2a-4038-4d24-9bb0-856aecc50dda" (UID: "d1040b2a-4038-4d24-9bb0-856aecc50dda"). InnerVolumeSpecName "kube-api-access-9jwl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:20.778066 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.777997    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4" pod "c6aedbee-5907-4b9e-b548-19f65dd1314d" (UID: "c6aedbee-5907-4b9e-b548-19f65dd1314d"). InnerVolumeSpecName "kube-api-access-2tzn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:20.876908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.876869    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aedbee-5907-4b9e-b548-19f65dd1314d-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:20.876908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.876902    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1040b2a-4038-4d24-9bb0-856aecc50dda-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:20.877136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.876916    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jwl4\" (UniqueName: \"kubernetes.io/projected/d1040b2a-4038-4d24-9bb0-856aecc50dda-kube-api-access-9jwl4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:20.877136 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:20.876929    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzn4\" (UniqueName: \"kubernetes.io/projected/c6aedbee-5907-4b9e-b548-19f65dd1314d-kube-api-access-2tzn4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:21.063922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.063877    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:21.063922 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.063902    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" podUID="9fbf6674-283c-43de-810e-94f436247a5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:21.424371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.424293    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"
Oct 30 12:45:21.424371 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.424337    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc" event={"ID":"c6aedbee-5907-4b9e-b548-19f65dd1314d","Type":"ContainerDied","Data":"2043daeeecacb21613b87eaba3449fa8cdd32cab71de3b95ec2d2621dfa5a68c"}
Oct 30 12:45:21.424945 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.424390    5060 scope.go:122] "RemoveContainer" containerID="1cbfbab593eb553cdec24ddf3fbf6ae5ecfe058d80d6e9cb22b90bd6648b2fb2"
Oct 30 12:45:21.426003 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.425979    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5" event={"ID":"d1040b2a-4038-4d24-9bb0-856aecc50dda","Type":"ContainerDied","Data":"187ebcb4dc4b0c34662d62ce1a7fc44da266e6f437bef2b1656ab09853f1165a"}
Oct 30 12:45:21.426166 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.426132    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"
Oct 30 12:45:21.439363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.439338    5060 scope.go:122] "RemoveContainer" containerID="de0b5ae46bb8ebf29a831bfcd5e2f8d786132d66a755e323d93cb855831e67a7"
Oct 30 12:45:21.524080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.524017    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"]
Oct 30 12:45:21.529414 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.529393    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889405-whzc5"]
Oct 30 12:45:21.594950 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.594877    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:21.611874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.611846    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:21.617689 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.617619    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:21.647701 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.647678    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"]
Oct 30 12:45:21.651301 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.651276    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889405-gsmhc"]
Oct 30 12:45:21.687422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.687357    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca\") pod \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") "
Oct 30 12:45:21.687422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.687406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn\" (UniqueName: \"kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn\") pod \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") "
Oct 30 12:45:21.687625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.687532    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs\" (UniqueName: \"kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs\") pod \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\" (UID: \"d1263bb2-a207-4313-bdfa-1ec1a96b1b25\") "
Oct 30 12:45:21.687705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.687679    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca" pod "d1263bb2-a207-4313-bdfa-1ec1a96b1b25" (UID: "d1263bb2-a207-4313-bdfa-1ec1a96b1b25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:21.687966 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.687945    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:21.689771 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.689747    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn" pod "89785f16-1a3f-48d0-b3dc-21bb8f40d175" (UID: "89785f16-1a3f-48d0-b3dc-21bb8f40d175"). InnerVolumeSpecName "kube-api-access-79hhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:21.690205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.690184    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs" pod "d1263bb2-a207-4313-bdfa-1ec1a96b1b25" (UID: "d1263bb2-a207-4313-bdfa-1ec1a96b1b25"). InnerVolumeSpecName "kube-api-access-dk5qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:21.763146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.763107    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:21.788981 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.788944    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca\") pod \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\" (UID: \"89785f16-1a3f-48d0-b3dc-21bb8f40d175\") "
Oct 30 12:45:21.789368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.789345    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5qs\" (UniqueName: \"kubernetes.io/projected/d1263bb2-a207-4313-bdfa-1ec1a96b1b25-kube-api-access-dk5qs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:21.789368 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.789355    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca" pod "89785f16-1a3f-48d0-b3dc-21bb8f40d175" (UID: "89785f16-1a3f-48d0-b3dc-21bb8f40d175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:21.789564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.789372    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79hhn\" (UniqueName: \"kubernetes.io/projected/89785f16-1a3f-48d0-b3dc-21bb8f40d175-kube-api-access-79hhn\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:21.890753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:21.890722    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89785f16-1a3f-48d0-b3dc-21bb8f40d175-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:22.035666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.035633    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"]
Oct 30 12:45:22.035932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.035881    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy" containerID="cri-o://69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594" gracePeriod=30
Oct 30 12:45:22.144672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.140916    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:22.335367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.335295    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" path="/var/lib/kubelet/pods/c6aedbee-5907-4b9e-b548-19f65dd1314d/volumes"
Oct 30 12:45:22.335781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.335767    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" path="/var/lib/kubelet/pods/d1040b2a-4038-4d24-9bb0-856aecc50dda/volumes"
Oct 30 12:45:22.440759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.440731    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"
Oct 30 12:45:22.441249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.440730    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64" event={"ID":"d1263bb2-a207-4313-bdfa-1ec1a96b1b25","Type":"ContainerDied","Data":"abf24dc3fe4632d025c59c82edd0d9689b1ce4cc73e84eb9d67d91a265d07ed4"}
Oct 30 12:45:22.441249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.440838    5060 scope.go:122] "RemoveContainer" containerID="5ffd019bc91fe739936e7ba0005ee0a4d99bbff6300f4721e313c4fe93159183"
Oct 30 12:45:22.442536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.442508    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"
Oct 30 12:45:22.442659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.442558    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d" event={"ID":"89785f16-1a3f-48d0-b3dc-21bb8f40d175","Type":"ContainerDied","Data":"25f6050d5e0d67ab832d5ae56fe68375189844bc8e962db21fb6aacdde1064c2"}
Oct 30 12:45:22.452338 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.452321    5060 scope.go:122] "RemoveContainer" containerID="a55d03a0ddc1a7d7f188385a2c51be444cb3cfb2c51c1d3cf7635640fd8bbf1c"
Oct 30 12:45:22.504875 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.504851    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"]
Oct 30 12:45:22.508389 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.508367    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889405-ccl64"]
Oct 30 12:45:22.545782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.545755    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"]
Oct 30 12:45:22.549963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.549941    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889405-b4w2d"]
Oct 30 12:45:22.665214 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.665171    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"]
Oct 30 12:45:22.666006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.665984    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.666006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666006    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666021    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666027    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666039    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666046    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666055    5060 cpu_manager.go:414] "containerMap: removing container" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666060    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666075    5060 cpu_manager.go:414] "containerMap: removing container" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666080    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666089    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666095    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666105    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666110    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666117    5060 cpu_manager.go:414] "containerMap: removing container" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666121    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666130    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666135    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666144    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666148    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666165    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666170    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666182    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.666182 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666187    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666200    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666206    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666292    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666300    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666308    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666315    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666323    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666329    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666336    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666343    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666347    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666353    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666358    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666364    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666369    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6aedbee-5907-4b9e-b548-19f65dd1314d" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666377    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666383    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1040b2a-4038-4d24-9bb0-856aecc50dda" containerName="cleanup"
Oct 30 12:45:22.667026 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.666390    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerName="cleanup"
Oct 30 12:45:22.669993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.669970    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.690495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.687690    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:45:22.690495 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.688386    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:45:22.692382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.692359    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"]
Oct 30 12:45:22.700975 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.700943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.701117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.701007    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.701117 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.701087    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvmv\" (UniqueName: \"kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.748744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.748708    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:45:22.748928 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.748765    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:45:22.802469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.801976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.802469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.802031    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.802469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.802069    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvmv\" (UniqueName: \"kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.803261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.803236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.806319 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.806270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.831493 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.826642    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvmv\" (UniqueName: \"kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:22.872795 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.872774    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:45:22.902694 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902661    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.902854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902735    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.902854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902770    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.902854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902799    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.902854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902828    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.903096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902903    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.903096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902952    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.903096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.902994    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.903096 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903032    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs\") pod \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\" (UID: \"cb182d45-f45a-46c8-a9b4-0d2d97113b72\") "
Oct 30 12:45:22.903317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903091    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:22.903317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903109    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:22.903317 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903292    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:22.903515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903478    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:22.903581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903559    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:22.903581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903576    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:22.903703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903587    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:22.903703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903595    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:22.903703 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.903652    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:22.905259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.905225    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:45:22.905668 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.905648    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:22.905801 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.905783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:22.905866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.905785    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x" pod "cb182d45-f45a-46c8-a9b4-0d2d97113b72" (UID: "cb182d45-f45a-46c8-a9b4-0d2d97113b72"). InnerVolumeSpecName "kube-api-access-r276x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:22.988990 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:22.988958    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:23.004442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.004413    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:23.004442 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.004438    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r276x\" (UniqueName: \"kubernetes.io/projected/cb182d45-f45a-46c8-a9b4-0d2d97113b72-kube-api-access-r276x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:23.004631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.004447    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:23.004631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.004478    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:23.004631 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.004490    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb182d45-f45a-46c8-a9b4-0d2d97113b72-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:23.294218 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:23.294187    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aeb137_5f71_4533_9406_c1d7395adfea.slice/crio-da55a6bad76922425b3ea17ea6be94fe511089ec4cb0a5ed791774e418d836f6 WatchSource:0}: Error finding container da55a6bad76922425b3ea17ea6be94fe511089ec4cb0a5ed791774e418d836f6: Status 404 returned error can't find the container with id da55a6bad76922425b3ea17ea6be94fe511089ec4cb0a5ed791774e418d836f6
Oct 30 12:45:23.297403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.297375    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"]
Oct 30 12:45:23.454093 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.454058    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx" event={"ID":"33aeb137-5f71-4533-9406-c1d7395adfea","Type":"ContainerStarted","Data":"da55a6bad76922425b3ea17ea6be94fe511089ec4cb0a5ed791774e418d836f6"}
Oct 30 12:45:23.458151 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.458125    5060 generic.go:356] "Generic (PLEG): container finished" podID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerID="5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce" exitCode=0
Oct 30 12:45:23.458274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.458177    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" event={"ID":"cb182d45-f45a-46c8-a9b4-0d2d97113b72","Type":"ContainerDied","Data":"5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce"}
Oct 30 12:45:23.458274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.458254    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h" event={"ID":"cb182d45-f45a-46c8-a9b4-0d2d97113b72","Type":"ContainerDied","Data":"997dbda2dc5754df4fe47ac56a05c6454f63d5355f3dcae3df6657a138095540"}
Oct 30 12:45:23.458364 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.458201    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"
Oct 30 12:45:23.458408 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.458285    5060 scope.go:122] "RemoveContainer" containerID="5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce"
Oct 30 12:45:23.472391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.472372    5060 scope.go:122] "RemoveContainer" containerID="d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"
Oct 30 12:45:23.484762 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.484735    5060 scope.go:122] "RemoveContainer" containerID="5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce"
Oct 30 12:45:23.485108 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:23.485082    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce\": container with ID starting with 5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce not found: ID does not exist" containerID="5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce"
Oct 30 12:45:23.485192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.485122    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce"} err="failed to get container status \"5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce\": rpc error: code = NotFound desc = could not find container \"5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce\": container with ID starting with 5c36e19e340b1d62b76e1e20bd98e6818bbc0af77ab981b7ed1e5a1a5cb098ce not found: ID does not exist"
Oct 30 12:45:23.485192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.485147    5060 scope.go:122] "RemoveContainer" containerID="d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"
Oct 30 12:45:23.485601 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:23.485571    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17\": container with ID starting with d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17 not found: ID does not exist" containerID="d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"
Oct 30 12:45:23.485734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.485609    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17"} err="failed to get container status \"d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17\": rpc error: code = NotFound desc = could not find container \"d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17\": container with ID starting with d9745451e7d8221eb14a0e388356bebcdbbabe724e7a12f49ee4a73dc8549e17 not found: ID does not exist"
Oct 30 12:45:23.515839 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.515802    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"]
Oct 30 12:45:23.521197 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.521169    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-87f2e6-openshift-default-59d48f5df9-sff2h"]
Oct 30 12:45:23.560715 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:23.560644    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:24.032114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.032080    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"]
Oct 30 12:45:24.035366 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.035342    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ed7099-nbrph"]
Oct 30 12:45:24.040391 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.040355    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" podUID="88864962-1518-4fec-82dc-a65b4afe8214" containerName="payload-processing" containerID="cri-o://bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272" gracePeriod=25
Oct 30 12:45:24.087165 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.087130    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerName="payload-pre-processing" containerID="cri-o://e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1" gracePeriod=25
Oct 30 12:45:24.336323 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.336245    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89785f16-1a3f-48d0-b3dc-21bb8f40d175" path="/var/lib/kubelet/pods/89785f16-1a3f-48d0-b3dc-21bb8f40d175/volumes"
Oct 30 12:45:24.336766 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.336751    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" path="/var/lib/kubelet/pods/cb182d45-f45a-46c8-a9b4-0d2d97113b72/volumes"
Oct 30 12:45:24.337099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.337087    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" path="/var/lib/kubelet/pods/d1263bb2-a207-4313-bdfa-1ec1a96b1b25/volumes"
Oct 30 12:45:24.337799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.337784    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d3b44f-0152-4ccf-90db-323541ed1c02" path="/var/lib/kubelet/pods/f9d3b44f-0152-4ccf-90db-323541ed1c02/volumes"
Oct 30 12:45:24.470761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.470723    5060 generic.go:356] "Generic (PLEG): container finished" podID="33aeb137-5f71-4533-9406-c1d7395adfea" containerID="aa389126221395bef2ca659e7d43275c765434a69247182d12db5066dab38fa5" exitCode=0
Oct 30 12:45:24.471160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:24.470792    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx" event={"ID":"33aeb137-5f71-4533-9406-c1d7395adfea","Type":"ContainerDied","Data":"aa389126221395bef2ca659e7d43275c765434a69247182d12db5066dab38fa5"}
Oct 30 12:45:25.576476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.575920    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:25.652976 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.652951    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:25.737025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.736948    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca\") pod \"33aeb137-5f71-4533-9406-c1d7395adfea\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") "
Oct 30 12:45:25.737185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.737092    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv\" (UniqueName: \"kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv\") pod \"33aeb137-5f71-4533-9406-c1d7395adfea\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") "
Oct 30 12:45:25.737185 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.737118    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca\") pod \"33aeb137-5f71-4533-9406-c1d7395adfea\" (UID: \"33aeb137-5f71-4533-9406-c1d7395adfea\") "
Oct 30 12:45:25.737314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.737285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca" pod "33aeb137-5f71-4533-9406-c1d7395adfea" (UID: "33aeb137-5f71-4533-9406-c1d7395adfea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:25.739300 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.739270    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv" pod "33aeb137-5f71-4533-9406-c1d7395adfea" (UID: "33aeb137-5f71-4533-9406-c1d7395adfea"). InnerVolumeSpecName "kube-api-access-nhvmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:25.739433 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.739318    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca" pod "33aeb137-5f71-4533-9406-c1d7395adfea" (UID: "33aeb137-5f71-4533-9406-c1d7395adfea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:45:25.838895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.838858    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhvmv\" (UniqueName: \"kubernetes.io/projected/33aeb137-5f71-4533-9406-c1d7395adfea-kube-api-access-nhvmv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:25.838895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.838891    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:25.838895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:25.838900    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aeb137-5f71-4533-9406-c1d7395adfea-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:26.492090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:26.492059    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"
Oct 30 12:45:26.492090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:26.492081    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx" event={"ID":"33aeb137-5f71-4533-9406-c1d7395adfea","Type":"ContainerDied","Data":"da55a6bad76922425b3ea17ea6be94fe511089ec4cb0a5ed791774e418d836f6"}
Oct 30 12:45:26.492312 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:26.492142    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da55a6bad76922425b3ea17ea6be94fe511089ec4cb0a5ed791774e418d836f6"
Oct 30 12:45:27.321778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.321749    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:45:27.355083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355056    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355114    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355177    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355253 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355221    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355273    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355440 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355438    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355483    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:27.355652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355498    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy\") pod \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\" (UID: \"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde\") "
Oct 30 12:45:27.355652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:27.355652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355591    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:27.356037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355710    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:27.356037 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.355864    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:27.356230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.356169    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.356230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.356191    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.356230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.356207    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.356230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.356221    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.356601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.356234    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.357730 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.357699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:27.357859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.357784    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "kube-api-access-wp8bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:27.357859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.357782    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:27.357979 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.357854    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo" pod "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" (UID: "caf0fbc9-54b9-46d5-a4fe-04ed4943dbde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:45:27.359915 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.359884    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:27.457192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.457163    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp8bx\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-kube-api-access-wp8bx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.457192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.457191    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.457421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.457201    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.457421 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.457211    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:27.501642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.501591    5060 generic.go:356] "Generic (PLEG): container finished" podID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerID="69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594" exitCode=0
Oct 30 12:45:27.501831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.501646    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"
Oct 30 12:45:27.501831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.501685    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" event={"ID":"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde","Type":"ContainerDied","Data":"69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594"}
Oct 30 12:45:27.501831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.501763    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4" event={"ID":"caf0fbc9-54b9-46d5-a4fe-04ed4943dbde","Type":"ContainerDied","Data":"9153c9d54f622f4162ef21deebfcab81d5dd8f82831960ee97232c130a2b7e1c"}
Oct 30 12:45:27.501831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.501795    5060 scope.go:122] "RemoveContainer" containerID="69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594"
Oct 30 12:45:27.518696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.516612    5060 scope.go:122] "RemoveContainer" containerID="69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594"
Oct 30 12:45:27.518696 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:27.517055    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594\": container with ID starting with 69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594 not found: ID does not exist" containerID="69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594"
Oct 30 12:45:27.518696 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.517093    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594"} err="failed to get container status \"69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594\": rpc error: code = NotFound desc = could not find container \"69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594\": container with ID starting with 69232d9ed7501059a4dc3e06ec737cc0be5dab939be7bb86c8996d0987be8594 not found: ID does not exist"
Oct 30 12:45:27.559615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.559531    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"]
Oct 30 12:45:27.569517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:27.569476    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-gdlz4"]
Oct 30 12:45:28.334567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:28.334531    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" path="/var/lib/kubelet/pods/caf0fbc9-54b9-46d5-a4fe-04ed4943dbde/volumes"
Oct 30 12:45:28.844592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:28.844547    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:28.985665 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:28.985618    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.347782 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:29.347723    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:30.070958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.070930    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:45:30.110991 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.110962    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:45:30.195946 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.195907    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume\") pod \"9fbf6674-283c-43de-810e-94f436247a5a\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") "
Oct 30 12:45:30.196158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.195956    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2\" (UniqueName: \"kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2\") pod \"63f28db7-7ed3-42bd-b638-ebb40980903a\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") "
Oct 30 12:45:30.196158 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.196094    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume\") pod \"63f28db7-7ed3-42bd-b638-ebb40980903a\" (UID: \"63f28db7-7ed3-42bd-b638-ebb40980903a\") "
Oct 30 12:45:30.196292 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.196162    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss\" (UniqueName: \"kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss\") pod \"9fbf6674-283c-43de-810e-94f436247a5a\" (UID: \"9fbf6674-283c-43de-810e-94f436247a5a\") "
Oct 30 12:45:30.196372 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.196335    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume" pod "9fbf6674-283c-43de-810e-94f436247a5a" (UID: "9fbf6674-283c-43de-810e-94f436247a5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:30.196535 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.196512    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume" pod "63f28db7-7ed3-42bd-b638-ebb40980903a" (UID: "63f28db7-7ed3-42bd-b638-ebb40980903a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:30.196693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.196664    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbf6674-283c-43de-810e-94f436247a5a-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:30.196693 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.196691    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f28db7-7ed3-42bd-b638-ebb40980903a-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:30.198726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.198702    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss" pod "9fbf6674-283c-43de-810e-94f436247a5a" (UID: "9fbf6674-283c-43de-810e-94f436247a5a"). InnerVolumeSpecName "kube-api-access-58fss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:30.198828 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.198813    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2" pod "63f28db7-7ed3-42bd-b638-ebb40980903a" (UID: "63f28db7-7ed3-42bd-b638-ebb40980903a"). InnerVolumeSpecName "kube-api-access-n9kc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:30.297602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.297513    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58fss\" (UniqueName: \"kubernetes.io/projected/9fbf6674-283c-43de-810e-94f436247a5a-kube-api-access-58fss\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:30.297602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.297541    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9kc2\" (UniqueName: \"kubernetes.io/projected/63f28db7-7ed3-42bd-b638-ebb40980903a-kube-api-access-n9kc2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:30.533640 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.533604    5060 generic.go:356] "Generic (PLEG): container finished" podID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerID="a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8" exitCode=0
Oct 30 12:45:30.534114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.533658    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"
Oct 30 12:45:30.534114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.533692    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" event={"ID":"63f28db7-7ed3-42bd-b638-ebb40980903a","Type":"ContainerDied","Data":"a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8"}
Oct 30 12:45:30.534114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.533748    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz" event={"ID":"63f28db7-7ed3-42bd-b638-ebb40980903a","Type":"ContainerDied","Data":"59477beb92850c7d2d5ce6d7acf153a19654951d754404b4f113740d4de69ef3"}
Oct 30 12:45:30.534114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.533771    5060 scope.go:122] "RemoveContainer" containerID="a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8"
Oct 30 12:45:30.535592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.535511    5060 generic.go:356] "Generic (PLEG): container finished" podID="9fbf6674-283c-43de-810e-94f436247a5a" containerID="916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8" exitCode=0
Oct 30 12:45:30.535592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.535551    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"
Oct 30 12:45:30.535592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.535600    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" event={"ID":"9fbf6674-283c-43de-810e-94f436247a5a","Type":"ContainerDied","Data":"916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8"}
Oct 30 12:45:30.535870 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.535649    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq" event={"ID":"9fbf6674-283c-43de-810e-94f436247a5a","Type":"ContainerDied","Data":"5c913288678e83c9452ad663fbf81c26bcba6752a281b52266825c2aac379f83"}
Oct 30 12:45:30.547233 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.547209    5060 scope.go:122] "RemoveContainer" containerID="a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8"
Oct 30 12:45:30.547567 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:30.547545    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8\": container with ID starting with a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8 not found: ID does not exist" containerID="a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8"
Oct 30 12:45:30.547700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.547572    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8"} err="failed to get container status \"a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8\": rpc error: code = NotFound desc = could not find container \"a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8\": container with ID starting with a9ad8cb58829222852ddcdf59648dbcc343ee001be479f57f1845d01fe8649e8 not found: ID does not exist"
Oct 30 12:45:30.547700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.547590    5060 scope.go:122] "RemoveContainer" containerID="916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8"
Oct 30 12:45:30.557325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.557305    5060 scope.go:122] "RemoveContainer" containerID="916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8"
Oct 30 12:45:30.557629 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:30.557610    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8\": container with ID starting with 916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8 not found: ID does not exist" containerID="916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8"
Oct 30 12:45:30.557690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.557641    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8"} err="failed to get container status \"916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8\": rpc error: code = NotFound desc = could not find container \"916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8\": container with ID starting with 916950aaba1abd975bf9c1f395897c6505843098b7168c72b36c43d3095e35b8 not found: ID does not exist"
Oct 30 12:45:30.586960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.586932    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"]
Oct 30 12:45:30.601074 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.601043    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f1b14f-f5b65c94c-vgldq"]
Oct 30 12:45:30.627570 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.627541    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"]
Oct 30 12:45:30.631101 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:30.631079    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f1b14f-587988bb5b-x2cqz"]
Oct 30 12:45:31.039392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:31.039358    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"]
Oct 30 12:45:31.042538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:31.042512    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"]
Oct 30 12:45:31.045397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:31.045376    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db7b6d7-fdpg4"]
Oct 30 12:45:31.050855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:31.050834    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8823713-lmr9q"]
Oct 30 12:45:31.170033 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:31.169997    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:32.338260 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.338216    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" path="/var/lib/kubelet/pods/63f28db7-7ed3-42bd-b638-ebb40980903a/volumes"
Oct 30 12:45:32.338985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.338958    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5e6b5a-3fe3-47bd-948f-171587193952" path="/var/lib/kubelet/pods/6b5e6b5a-3fe3-47bd-948f-171587193952/volumes"
Oct 30 12:45:32.339825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.339804    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956f1d85-96a2-4dbb-a595-866c71f49724" path="/var/lib/kubelet/pods/956f1d85-96a2-4dbb-a595-866c71f49724/volumes"
Oct 30 12:45:32.341008 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.340986    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbf6674-283c-43de-810e-94f436247a5a" path="/var/lib/kubelet/pods/9fbf6674-283c-43de-810e-94f436247a5a/volumes"
Oct 30 12:45:32.646550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.646439    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:32.646550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.646512    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" podUID="88864962-1518-4fec-82dc-a65b4afe8214" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:32.680476 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:32.680408    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:33.169606 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:33.169568    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"]
Oct 30 12:45:33.244585 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:33.244541    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:33.398162 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:33.398129    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:33.905838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:33.905799    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:34.031955 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:34.031916    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"]
Oct 30 12:45:34.043840 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:34.043807    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9064398b-crzbq"]
Oct 30 12:45:34.335742 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:34.335707    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbefc45-82a8-4bba-b93d-05b8e8e1504d" path="/var/lib/kubelet/pods/adbefc45-82a8-4bba-b93d-05b8e8e1504d/volumes"
Oct 30 12:45:35.320328 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:35.320287    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:37.057446 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:37.057356    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:38.170847 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:38.170804    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerName="payload-pre-processing" containerID="cri-o://821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244" gracePeriod=25
Oct 30 12:45:38.509381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:38.509346    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:38.767649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:38.767557    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:38.915406 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:38.915370    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.060051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.059944    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.130461 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.130421    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"]
Oct 30 12:45:39.142521 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.142495    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4fff4f-8hnjx"]
Oct 30 12:45:39.237473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.237402    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.568744 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.568705    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.719088 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.719053    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.864868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:39.864786    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:40.194707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:40.194620    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:40.335838 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:40.335799    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33aeb137-5f71-4533-9406-c1d7395adfea" path="/var/lib/kubelet/pods/33aeb137-5f71-4533-9406-c1d7395adfea/volumes"
Oct 30 12:45:40.865590 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:40.865552    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:41.013721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.013681    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:41.126530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.126418    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"]
Oct 30 12:45:41.126755 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.126722    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy" containerID="cri-o://6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a" gracePeriod=30
Oct 30 12:45:41.204898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.204841    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:41.661350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.661319    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:41.738863 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.738828    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"]
Oct 30 12:45:41.739303 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739290    5060 cpu_manager.go:414] "containerMap: removing container" podUID="33aeb137-5f71-4533-9406-c1d7395adfea" containerName="revoke-keys"
Oct 30 12:45:41.739355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739305    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33aeb137-5f71-4533-9406-c1d7395adfea" containerName="revoke-keys"
Oct 30 12:45:41.739355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739323    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" containerName="cleanup"
Oct 30 12:45:41.739355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739328    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe88a80-a679-428d-a502-f1e1003dc619" containerName="cleanup"
Oct 30 12:45:41.739355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739339    5060 cpu_manager.go:414] "containerMap: removing container" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy"
Oct 30 12:45:41.739355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739344    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy"
Oct 30 12:45:41.739355 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739351    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739358    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1263bb2-a207-4313-bdfa-1ec1a96b1b25" containerName="cleanup"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739366    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739371    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739379    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9fbf6674-283c-43de-810e-94f436247a5a" containerName="payload-pre-processing"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739383    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbf6674-283c-43de-810e-94f436247a5a" containerName="payload-pre-processing"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739393    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739397    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739404    5060 cpu_manager.go:414] "containerMap: removing container" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerName="payload-processing"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739409    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerName="payload-processing"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739421    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerName="cleanup"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739425    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca7cb7f0-4da4-4b96-84c7-6e3a0fcf236a" containerName="cleanup"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739533    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="33aeb137-5f71-4533-9406-c1d7395adfea" containerName="revoke-keys"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739545    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf0fbc9-54b9-46d5-a4fe-04ed4943dbde" containerName="istio-proxy"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739551    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739557    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbf6674-283c-43de-810e-94f436247a5a" containerName="payload-pre-processing"
Oct 30 12:45:41.739608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.739564    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f28db7-7ed3-42bd-b638-ebb40980903a" containerName="payload-processing"
Oct 30 12:45:41.743503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.743482    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.758690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.758664    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:45:41.758848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.758817    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:45:41.759786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.759766    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"]
Oct 30 12:45:41.778332 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.778309    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:45:41.778506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.778346    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:45:41.852707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.852673    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.852850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.852737    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.852905 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.852882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvkgs\" (UniqueName: \"kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.954006 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.953979    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvkgs\" (UniqueName: \"kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.954178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.954035    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.954178 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.954080    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.954686 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.954658    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.956602 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.956572    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:41.977798 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:41.977775    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvkgs\" (UniqueName: \"kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs\") pod \"maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:42.058013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:42.057982    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:42.366646 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:42.366602    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c3f67c_75d0_4e79_b580_ff86d9404ebb.slice/crio-f7163c508813429f9b43b53d597fe9bb7121c23bf0f434b321bc01c0216e37e2 WatchSource:0}: Error finding container f7163c508813429f9b43b53d597fe9bb7121c23bf0f434b321bc01c0216e37e2: Status 404 returned error can't find the container with id f7163c508813429f9b43b53d597fe9bb7121c23bf0f434b321bc01c0216e37e2
Oct 30 12:45:42.367774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:42.367749    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"]
Oct 30 12:45:42.646344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:42.646276    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" podUID="88864962-1518-4fec-82dc-a65b4afe8214" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:42.646344 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:42.646293    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:42.660792 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:42.660758    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m" event={"ID":"76c3f67c-75d0-4e79-b580-ff86d9404ebb","Type":"ContainerStarted","Data":"f7163c508813429f9b43b53d597fe9bb7121c23bf0f434b321bc01c0216e37e2"}
Oct 30 12:45:43.464043 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:43.463957    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:43.669395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:43.669361    5060 generic.go:356] "Generic (PLEG): container finished" podID="76c3f67c-75d0-4e79-b580-ff86d9404ebb" containerID="4e66e2105ad57feb18e4d9c2e4f08d91392a8e1d6c2dacad5ac982d41568cb43" exitCode=0
Oct 30 12:45:43.669619 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:43.669470    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m" event={"ID":"76c3f67c-75d0-4e79-b580-ff86d9404ebb","Type":"ContainerDied","Data":"4e66e2105ad57feb18e4d9c2e4f08d91392a8e1d6c2dacad5ac982d41568cb43"}
Oct 30 12:45:44.356133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.356104    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:45:44.382296 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.382272    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2\" (UniqueName: \"kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2\") pod \"88864962-1518-4fec-82dc-a65b4afe8214\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") "
Oct 30 12:45:44.382392 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.382362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume\") pod \"88864962-1518-4fec-82dc-a65b4afe8214\" (UID: \"88864962-1518-4fec-82dc-a65b4afe8214\") "
Oct 30 12:45:44.382757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.382737    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume" pod "88864962-1518-4fec-82dc-a65b4afe8214" (UID: "88864962-1518-4fec-82dc-a65b4afe8214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:44.384684 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.384664    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2" pod "88864962-1518-4fec-82dc-a65b4afe8214" (UID: "88864962-1518-4fec-82dc-a65b4afe8214"). InnerVolumeSpecName "kube-api-access-957q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:44.401131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.401090    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:45:44.483826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.483783    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg\" (UniqueName: \"kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg\") pod \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") "
Oct 30 12:45:44.484324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.483851    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume\") pod \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\" (UID: \"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac\") "
Oct 30 12:45:44.484324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.484293    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume" pod "0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" (UID: "0eaec9e2-0e07-4fbe-94ad-462f1361e5ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:44.484324 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.484303    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88864962-1518-4fec-82dc-a65b4afe8214-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:44.484500 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.484344    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957q2\" (UniqueName: \"kubernetes.io/projected/88864962-1518-4fec-82dc-a65b4afe8214-kube-api-access-957q2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:44.486262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.486240    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg" pod "0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" (UID: "0eaec9e2-0e07-4fbe-94ad-462f1361e5ac"). InnerVolumeSpecName "kube-api-access-pg2dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:44.585690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.585608    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:44.585690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.585640    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2dg\" (UniqueName: \"kubernetes.io/projected/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac-kube-api-access-pg2dg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:44.680016 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.679979    5060 generic.go:356] "Generic (PLEG): container finished" podID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerID="e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1" exitCode=0
Oct 30 12:45:44.680196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.680027    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"
Oct 30 12:45:44.680196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.680066    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" event={"ID":"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac","Type":"ContainerDied","Data":"e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1"}
Oct 30 12:45:44.680196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.680123    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc" event={"ID":"0eaec9e2-0e07-4fbe-94ad-462f1361e5ac","Type":"ContainerDied","Data":"e9d11f05e1576c10e150d6a2d831451915f1cace4ef35b31a175e5283931e88a"}
Oct 30 12:45:44.680196 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.680153    5060 scope.go:122] "RemoveContainer" containerID="e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1"
Oct 30 12:45:44.682280 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.682252    5060 generic.go:356] "Generic (PLEG): container finished" podID="88864962-1518-4fec-82dc-a65b4afe8214" containerID="bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272" exitCode=0
Oct 30 12:45:44.682659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.682639    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"
Oct 30 12:45:44.686080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.685604    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" event={"ID":"88864962-1518-4fec-82dc-a65b4afe8214","Type":"ContainerDied","Data":"bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272"}
Oct 30 12:45:44.686080 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.685657    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl" event={"ID":"88864962-1518-4fec-82dc-a65b4afe8214","Type":"ContainerDied","Data":"aa026732bd36099bf6dd28c5c78cbb98ab0f2708e4785fb1b340c788c23885ba"}
Oct 30 12:45:44.693125 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.693102    5060 scope.go:122] "RemoveContainer" containerID="e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1"
Oct 30 12:45:44.693481 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:44.693434    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1\": container with ID starting with e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1 not found: ID does not exist" containerID="e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1"
Oct 30 12:45:44.693646 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.693619    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1"} err="failed to get container status \"e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1\": rpc error: code = NotFound desc = could not find container \"e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1\": container with ID starting with e71a026453f6a9b8df7c9f563705a6d6518971d9f7638b79c3762dc3ac166bd1 not found: ID does not exist"
Oct 30 12:45:44.693763 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.693743    5060 scope.go:122] "RemoveContainer" containerID="bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272"
Oct 30 12:45:44.707273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.707253    5060 scope.go:122] "RemoveContainer" containerID="bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272"
Oct 30 12:45:44.707665 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:44.707635    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272\": container with ID starting with bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272 not found: ID does not exist" containerID="bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272"
Oct 30 12:45:44.707758 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.707679    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272"} err="failed to get container status \"bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272\": rpc error: code = NotFound desc = could not find container \"bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272\": container with ID starting with bcf580b9ca234f3d2e1c01f0a6cee787eb02280e95129169284f1554edca9272 not found: ID does not exist"
Oct 30 12:45:44.749753 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.749716    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"]
Oct 30 12:45:44.753908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.753875    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f8f132-76fb9bcd99-s2xkc"]
Oct 30 12:45:44.789987 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.789953    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"]
Oct 30 12:45:44.797002 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.796967    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f8f132-7fd7bc96cb-zs6nl"]
Oct 30 12:45:44.834396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.834372    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:44.891216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.891125    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs\" (UniqueName: \"kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs\") pod \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") "
Oct 30 12:45:44.891400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.891268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca\") pod \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") "
Oct 30 12:45:44.891400 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.891346    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca\") pod \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\" (UID: \"76c3f67c-75d0-4e79-b580-ff86d9404ebb\") "
Oct 30 12:45:44.891707 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.891684    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca" pod "76c3f67c-75d0-4e79-b580-ff86d9404ebb" (UID: "76c3f67c-75d0-4e79-b580-ff86d9404ebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:44.893706 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.893682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca" pod "76c3f67c-75d0-4e79-b580-ff86d9404ebb" (UID: "76c3f67c-75d0-4e79-b580-ff86d9404ebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:45:44.893825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.893782    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs" pod "76c3f67c-75d0-4e79-b580-ff86d9404ebb" (UID: "76c3f67c-75d0-4e79-b580-ff86d9404ebb"). InnerVolumeSpecName "kube-api-access-wvkgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:44.992192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.992157    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:44.992192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.992188    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c3f67c-75d0-4e79-b580-ff86d9404ebb-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:44.992398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:44.992205    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvkgs\" (UniqueName: \"kubernetes.io/projected/76c3f67c-75d0-4e79-b580-ff86d9404ebb-kube-api-access-wvkgs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:45.034497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:45.034442    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"]
Oct 30 12:45:45.037719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:45.037689    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabc5519-5cms2"]
Oct 30 12:45:45.264750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:45.264610    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:45.693804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:45.693715    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"
Oct 30 12:45:45.693804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:45.693733    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m" event={"ID":"76c3f67c-75d0-4e79-b580-ff86d9404ebb","Type":"ContainerDied","Data":"f7163c508813429f9b43b53d597fe9bb7121c23bf0f434b321bc01c0216e37e2"}
Oct 30 12:45:45.693804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:45.693778    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7163c508813429f9b43b53d597fe9bb7121c23bf0f434b321bc01c0216e37e2"
Oct 30 12:45:46.337708 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.337669    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" path="/var/lib/kubelet/pods/0eaec9e2-0e07-4fbe-94ad-462f1361e5ac/volumes"
Oct 30 12:45:46.338315 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.338290    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2fc0a2-cbfe-49ed-8941-89f04f7525c9" path="/var/lib/kubelet/pods/1f2fc0a2-cbfe-49ed-8941-89f04f7525c9/volumes"
Oct 30 12:45:46.338874 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.338852    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88864962-1518-4fec-82dc-a65b4afe8214" path="/var/lib/kubelet/pods/88864962-1518-4fec-82dc-a65b4afe8214/volumes"
Oct 30 12:45:46.406988 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.406927    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:45:46.409889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.409867    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.409968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.409911    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.409968 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.409937    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410006    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410045    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410078    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410129    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410159    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:46.410291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410186    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert\") pod \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\" (UID: \"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9\") "
Oct 30 12:45:46.410291 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410278    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:46.410626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410290    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:46.410626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410529    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:46.410626 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:46.410799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410624    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.410799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410644    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.410799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410661    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.410799 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.410674    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.412676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.412650    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:46.412793 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.412687    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "kube-api-access-kqbr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:46.412886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.412869    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:46.413013 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.412992    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo" pod "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" (UID: "65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:45:46.511353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.511316    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.511353 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.511353    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.511657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.511367    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbr4\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-kube-api-access-kqbr4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.511657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.511379    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.511657 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.511392    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:46.613921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.613879    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:46.709042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.709004    5060 generic.go:356] "Generic (PLEG): container finished" podID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerID="6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a" exitCode=0
Oct 30 12:45:46.709661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.709077    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"
Oct 30 12:45:46.709661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.709091    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" event={"ID":"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9","Type":"ContainerDied","Data":"6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a"}
Oct 30 12:45:46.709661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.709141    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6" event={"ID":"65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9","Type":"ContainerDied","Data":"34f91a5a45634b32c2a8b128c709551702fd16f174f3df3796509fcbf40434f1"}
Oct 30 12:45:46.709661 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.709167    5060 scope.go:122] "RemoveContainer" containerID="6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a"
Oct 30 12:45:46.721427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.721401    5060 scope.go:122] "RemoveContainer" containerID="7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"
Oct 30 12:45:46.732649 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.732626    5060 scope.go:122] "RemoveContainer" containerID="6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a"
Oct 30 12:45:46.732958 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:46.732936    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a\": container with ID starting with 6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a not found: ID does not exist" containerID="6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a"
Oct 30 12:45:46.733042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.732968    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a"} err="failed to get container status \"6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a\": rpc error: code = NotFound desc = could not find container \"6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a\": container with ID starting with 6ca59d4e10e3ecd290f56877cd633249da86d3729e4130fe6dd6ffeb61897e8a not found: ID does not exist"
Oct 30 12:45:46.733042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.732985    5060 scope.go:122] "RemoveContainer" containerID="7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"
Oct 30 12:45:46.733277 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:46.733256    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6\": container with ID starting with 7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6 not found: ID does not exist" containerID="7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"
Oct 30 12:45:46.733350 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.733284    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6"} err="failed to get container status \"7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6\": rpc error: code = NotFound desc = could not find container \"7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6\": container with ID starting with 7ee95bdd61fce034ae0e189156b43bedd522c736d618fef68c01407f507216f6 not found: ID does not exist"
Oct 30 12:45:46.747982 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.747947    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:46.762310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.762283    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"]
Oct 30 12:45:46.774967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.774940    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-mcjm6"]
Oct 30 12:45:46.894882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.894845    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"]
Oct 30 12:45:46.895110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.895082    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy" containerID="cri-o://ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99" gracePeriod=30
Oct 30 12:45:46.960790 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:46.960700    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:47.113261 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.113216    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:47.445202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.445120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:47.593678 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.593635    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:47.704354 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.704315    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"]
Oct 30 12:45:47.704938 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.704918    5060 cpu_manager.go:414] "containerMap: removing container" podUID="88864962-1518-4fec-82dc-a65b4afe8214" containerName="payload-processing"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.704941    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88864962-1518-4fec-82dc-a65b4afe8214" containerName="payload-processing"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.704966    5060 cpu_manager.go:414] "containerMap: removing container" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.704974    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.704993    5060 cpu_manager.go:414] "containerMap: removing container" podUID="76c3f67c-75d0-4e79-b580-ff86d9404ebb" containerName="revoke-keys"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705000    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c3f67c-75d0-4e79-b580-ff86d9404ebb" containerName="revoke-keys"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705018    5060 cpu_manager.go:414] "containerMap: removing container" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705025    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705059    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerName="payload-pre-processing"
Oct 30 12:45:47.705076 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705067    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerName="payload-pre-processing"
Oct 30 12:45:47.705625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705202    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eaec9e2-0e07-4fbe-94ad-462f1361e5ac" containerName="payload-pre-processing"
Oct 30 12:45:47.705625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705218    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb182d45-f45a-46c8-a9b4-0d2d97113b72" containerName="istio-proxy"
Oct 30 12:45:47.705625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705232    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="88864962-1518-4fec-82dc-a65b4afe8214" containerName="payload-processing"
Oct 30 12:45:47.705625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705243    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy"
Oct 30 12:45:47.705625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705254    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c3f67c-75d0-4e79-b580-ff86d9404ebb" containerName="revoke-keys"
Oct 30 12:45:47.705625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.705265    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" containerName="istio-proxy"
Oct 30 12:45:47.710282 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.710262    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.727201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.727175    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"]
Oct 30 12:45:47.727600 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.727574    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.727719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.727628    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.727719 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.727685    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqgg\" (UniqueName: \"kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.735774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.735750    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:45:47.735907 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.735876    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:45:47.735967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.735953    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:45:47.736083 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.736065    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:45:47.828336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.828288    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.828550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.828342    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.828550 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.828387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqgg\" (UniqueName: \"kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.829067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.829041    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.830962 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.830927    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.852530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.852502    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqgg\" (UniqueName: \"kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg\") pod \"maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:47.875617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:47.875590    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:48.030879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.030790    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:48.336245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.336216    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9" path="/var/lib/kubelet/pods/65ba23b6-ff8e-4f0f-ac5a-337e5770d6c9/volumes"
Oct 30 12:45:48.337283 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.337254    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:48.337852 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:45:48.337827    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fc9b59_dd58_493a_ae7a_96ef7fcd8e56.slice/crio-cc73314a5f4cb3cb09615ea30f9b3a5dceced3ed850682e2af76a1879500bc72 WatchSource:0}: Error finding container cc73314a5f4cb3cb09615ea30f9b3a5dceced3ed850682e2af76a1879500bc72: Status 404 returned error can't find the container with id cc73314a5f4cb3cb09615ea30f9b3a5dceced3ed850682e2af76a1879500bc72
Oct 30 12:45:48.338831 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.338814    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"]
Oct 30 12:45:48.735090 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.735047    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp" event={"ID":"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56","Type":"ContainerStarted","Data":"25d286afaf3afc8fa4b9ad24404046d405d2d7ead1a173cf907b8f0502e71085"}
Oct 30 12:45:48.735528 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.735147    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp" event={"ID":"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56","Type":"ContainerStarted","Data":"cc73314a5f4cb3cb09615ea30f9b3a5dceced3ed850682e2af76a1879500bc72"}
Oct 30 12:45:48.767810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.767772    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:48.791359 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:48.791316    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp" podStartSLOduration=1.791302223 podStartE2EDuration="1.791302223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:48.788915328 +0000 UTC m=+4552.583799822" watchObservedRunningTime="2026-10-30 12:45:48.791302223 +0000 UTC m=+4552.586186751"
Oct 30 12:45:49.059307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:49.059263    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:49.746309 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:49.746261    5060 generic.go:356] "Generic (PLEG): container finished" podID="c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" containerID="25d286afaf3afc8fa4b9ad24404046d405d2d7ead1a173cf907b8f0502e71085" exitCode=0
Oct 30 12:45:49.746734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:49.746405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp" event={"ID":"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56","Type":"ContainerDied","Data":"25d286afaf3afc8fa4b9ad24404046d405d2d7ead1a173cf907b8f0502e71085"}
Oct 30 12:45:49.858654 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:49.858620    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:45:50.896245 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.896212    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:50.942530 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.942496    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:50.966307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.966273    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca\") pod \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") "
Oct 30 12:45:50.966527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.966359    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg\" (UniqueName: \"kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg\") pod \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") "
Oct 30 12:45:50.966527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.966501    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca\") pod \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\" (UID: \"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56\") "
Oct 30 12:45:50.967833 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.967763    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca" pod "c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" (UID: "c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:50.970608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.970581    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca" pod "c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" (UID: "c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:45:50.970727 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:50.970636    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg" pod "c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" (UID: "c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56"). InnerVolumeSpecName "kube-api-access-mhqgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:51.068310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:51.068232    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:51.068310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:51.068260    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqgg\" (UniqueName: \"kubernetes.io/projected/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-kube-api-access-mhqgg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:51.068310 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:51.068272    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:51.765650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:51.765594    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"
Oct 30 12:45:51.765879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:51.765725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp" event={"ID":"c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56","Type":"ContainerDied","Data":"cc73314a5f4cb3cb09615ea30f9b3a5dceced3ed850682e2af76a1879500bc72"}
Oct 30 12:45:51.765879 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:51.765782    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc73314a5f4cb3cb09615ea30f9b3a5dceced3ed850682e2af76a1879500bc72"
Oct 30 12:45:51.795107 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:51.795081    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fc9b59_dd58_493a_ae7a_96ef7fcd8e56.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 12:45:52.168216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.168187    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:45:52.237479 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.236636    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"]
Oct 30 12:45:52.279112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279025    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279076    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279112 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279098    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279134    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279181    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279239    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279282    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279316    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279357    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert\") pod \"2b3efebc-c1e2-45ec-88e4-249d399d5194\" (UID: \"2b3efebc-c1e2-45ec-88e4-249d399d5194\") "
Oct 30 12:45:52.279441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279355    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:52.279906 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.279873    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:52.280129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.280099    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3efebc-c1e2-45ec-88e4-249d399d5194-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.280129 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.280117    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.280374 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.280354    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:52.280558 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.280538    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:52.280654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.280597    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:52.282262 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.282233    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:52.282517 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.282489    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "kube-api-access-2bdc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:52.282714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.282692    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:45:52.282894 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.282876    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo" pod "2b3efebc-c1e2-45ec-88e4-249d399d5194" (UID: "2b3efebc-c1e2-45ec-88e4-249d399d5194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:45:52.373642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.373613    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"]
Oct 30 12:45:52.381192 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381165    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"]
Oct 30 12:45:52.381594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381550    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.381594 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381581    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.381786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381596    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bdc7\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-kube-api-access-2bdc7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.381786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381612    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.381786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381626    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.381786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381639    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3efebc-c1e2-45ec-88e4-249d399d5194-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.381786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.381652    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3efebc-c1e2-45ec-88e4-249d399d5194-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:52.407654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.407621    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:52.564045 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.563956    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:52.736549 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.736517    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:52.774682 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.774646    5060 generic.go:356] "Generic (PLEG): container finished" podID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerID="ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99" exitCode=0
Oct 30 12:45:52.774853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.774714    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"
Oct 30 12:45:52.774853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.774723    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" event={"ID":"2b3efebc-c1e2-45ec-88e4-249d399d5194","Type":"ContainerDied","Data":"ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99"}
Oct 30 12:45:52.774853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.774777    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s" event={"ID":"2b3efebc-c1e2-45ec-88e4-249d399d5194","Type":"ContainerDied","Data":"a2aa7735bf051a41bbf0218537310267fe9993ae5c71684a8a8b00c359c29ecb"}
Oct 30 12:45:52.774853 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.774799    5060 scope.go:122] "RemoveContainer" containerID="ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99"
Oct 30 12:45:52.788432 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.788411    5060 scope.go:122] "RemoveContainer" containerID="ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99"
Oct 30 12:45:52.788847 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:52.788818    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99\": container with ID starting with ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99 not found: ID does not exist" containerID="ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99"
Oct 30 12:45:52.788993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.788856    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99"} err="failed to get container status \"ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99\": rpc error: code = NotFound desc = could not find container \"ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99\": container with ID starting with ea6478095c771818af6de6b5893034ff1af62c00b9b5de71b1d3ce76b4848c99 not found: ID does not exist"
Oct 30 12:45:52.949330 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.949295    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"]
Oct 30 12:45:52.959351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:52.959323    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-t4w4s"]
Oct 30 12:45:53.269012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:53.268919    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"]
Oct 30 12:45:53.284398 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:53.284368    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"]
Oct 30 12:45:53.387866 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:53.387812    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:54.334116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:54.334079    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" path="/var/lib/kubelet/pods/2b3efebc-c1e2-45ec-88e4-249d399d5194/volumes"
Oct 30 12:45:54.537650 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:54.537614    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:56.295156 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:56.295123    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:57.238021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:57.237974    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerName="payload-processing" containerID="cri-o://40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a" gracePeriod=25
Oct 30 12:45:57.346804 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:57.346771    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:57.940786 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:57.940750    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:58.339886 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.339847    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:58.456750 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.456725    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:45:58.481601 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.481570    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:58.546058 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.545983    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume\") pod \"2f3063e9-89dc-4de4-8412-575d2d6778d5\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") "
Oct 30 12:45:58.546215 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.546105    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj\" (UniqueName: \"kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj\") pod \"2f3063e9-89dc-4de4-8412-575d2d6778d5\" (UID: \"2f3063e9-89dc-4de4-8412-575d2d6778d5\") "
Oct 30 12:45:58.546395 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.546373    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume" pod "2f3063e9-89dc-4de4-8412-575d2d6778d5" (UID: "2f3063e9-89dc-4de4-8412-575d2d6778d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:58.548347 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.548322    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj" pod "2f3063e9-89dc-4de4-8412-575d2d6778d5" (UID: "2f3063e9-89dc-4de4-8412-575d2d6778d5"). InnerVolumeSpecName "kube-api-access-vgmtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:58.647685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.647641    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgmtj\" (UniqueName: \"kubernetes.io/projected/2f3063e9-89dc-4de4-8412-575d2d6778d5-kube-api-access-vgmtj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:58.647685 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.647686    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3063e9-89dc-4de4-8412-575d2d6778d5-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:45:58.837237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.837137    5060 generic.go:356] "Generic (PLEG): container finished" podID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerID="821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244" exitCode=0
Oct 30 12:45:58.837237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.837181    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"
Oct 30 12:45:58.837237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.837226    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" event={"ID":"2f3063e9-89dc-4de4-8412-575d2d6778d5","Type":"ContainerDied","Data":"821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244"}
Oct 30 12:45:58.837582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.837283    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs" event={"ID":"2f3063e9-89dc-4de4-8412-575d2d6778d5","Type":"ContainerDied","Data":"5b0bae501430942864aff3f412490e468aa5bc507708aabd477c304d33b9c3b6"}
Oct 30 12:45:58.837582 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.837310    5060 scope.go:122] "RemoveContainer" containerID="821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244"
Oct 30 12:45:58.850842 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.850822    5060 scope.go:122] "RemoveContainer" containerID="821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244"
Oct 30 12:45:58.851133 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:45:58.851112    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244\": container with ID starting with 821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244 not found: ID does not exist" containerID="821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244"
Oct 30 12:45:58.851189 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.851141    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244"} err="failed to get container status \"821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244\": rpc error: code = NotFound desc = could not find container \"821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244\": container with ID starting with 821c11a6f367633fd6a0236fa1fb2b71ead795de80ed663e59417cf3c7c58244 not found: ID does not exist"
Oct 30 12:45:58.891497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.891465    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"]
Oct 30 12:45:58.904057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:58.904032    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4fff4f-8cc4568d7-28cbs"]
Oct 30 12:45:59.133140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.133063    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:59.273497 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.273433    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:59.480592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.480550    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:59.827726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.827639    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"]
Oct 30 12:45:59.829700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.829675    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy"
Oct 30 12:45:59.829895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.829879    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy"
Oct 30 12:45:59.830024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830013    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerName="payload-pre-processing"
Oct 30 12:45:59.830110 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830097    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerName="payload-pre-processing"
Oct 30 12:45:59.830200 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830190    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" containerName="revoke-keys"
Oct 30 12:45:59.830279 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830270    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" containerName="revoke-keys"
Oct 30 12:45:59.830536 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830520    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" containerName="payload-pre-processing"
Oct 30 12:45:59.830659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830647    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" containerName="revoke-keys"
Oct 30 12:45:59.830761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.830752    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3efebc-c1e2-45ec-88e4-249d399d5194" containerName="istio-proxy"
Oct 30 12:45:59.836060 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.836040    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:45:59.854913 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.854882    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"]
Oct 30 12:45:59.860425 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.860400    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.861583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.861354    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.926827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.926789    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:45:59.926827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.926792    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:45:59.927437 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.927412    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"]
Oct 30 12:45:59.931489 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.931443    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:45:59.947255 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.947222    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"]
Oct 30 12:45:59.963057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.963032    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:45:59.963179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.963080    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57q4\" (UniqueName: \"kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:45:59.963179 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.963107    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:45:59.973041 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:45:59.973018    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:00.064106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064066    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.064106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064109    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57q4\" (UniqueName: \"kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.064331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064128    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.064331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.064331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064228    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsnv\" (UniqueName: \"kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.064331 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064259    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.064882 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.064864    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.066690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.066671    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.102469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.102315    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57q4\" (UniqueName: \"kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4\") pod \"maas-api-revoke-sub-4b7c329b-l48vk\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.117285 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.117245    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:00.165855 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.165818    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsnv\" (UniqueName: \"kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.166038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.165860    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.166038 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.165960    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.166576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.166549    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.168475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.168439    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.175072 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.175051    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"]
Oct 30 12:46:00.177131 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.177101    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:00.180202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.180178    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2bc4e2-98k7m"]
Oct 30 12:46:00.190593 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.190562    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsnv\" (UniqueName: \"kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv\") pod \"maas-api-revoke-sub-b0afeef5-kkk2n\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.247652 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.245881    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:00.314638 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.314600    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:00.339941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.339912    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3063e9-89dc-4de4-8412-575d2d6778d5" path="/var/lib/kubelet/pods/2f3063e9-89dc-4de4-8412-575d2d6778d5/volumes"
Oct 30 12:46:00.340329 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.340316    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c3f67c-75d0-4e79-b580-ff86d9404ebb" path="/var/lib/kubelet/pods/76c3f67c-75d0-4e79-b580-ff86d9404ebb/volumes"
Oct 30 12:46:00.464418 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.464389    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:00.499681 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.499647    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"]
Oct 30 12:46:00.575810 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:46:00.575783    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a84611_bc65_4e69_908c_8a04c92c629f.slice/crio-1fa6331b6dd8616d8f55f324e7d8be9492625acaecdf0d665a87d2ff9b22c7e1 WatchSource:0}: Error finding container 1fa6331b6dd8616d8f55f324e7d8be9492625acaecdf0d665a87d2ff9b22c7e1: Status 404 returned error can't find the container with id 1fa6331b6dd8616d8f55f324e7d8be9492625acaecdf0d665a87d2ff9b22c7e1
Oct 30 12:46:00.578191 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.578128    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"]
Oct 30 12:46:00.673599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.673562    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:00.871422 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.871381    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n" event={"ID":"c6a84611-bc65-4e69-908c-8a04c92c629f","Type":"ContainerStarted","Data":"1fa6331b6dd8616d8f55f324e7d8be9492625acaecdf0d665a87d2ff9b22c7e1"}
Oct 30 12:46:00.872773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:00.872736    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk" event={"ID":"baf26f18-a9ed-4385-ae09-956af9788fec","Type":"ContainerStarted","Data":"a717f24e8daf970105e77652adc812cb73a813c896e0e3f6a41c185eb06e9b55"}
Oct 30 12:46:01.881717 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:01.881637    5060 generic.go:356] "Generic (PLEG): container finished" podID="c6a84611-bc65-4e69-908c-8a04c92c629f" containerID="b85f8ac3bb3ba09589967d687a64bbc96c2b0ade5351265b79c57c7980a33b0f" exitCode=0
Oct 30 12:46:01.882116 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:01.881720    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n" event={"ID":"c6a84611-bc65-4e69-908c-8a04c92c629f","Type":"ContainerDied","Data":"b85f8ac3bb3ba09589967d687a64bbc96c2b0ade5351265b79c57c7980a33b0f"}
Oct 30 12:46:01.883608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:01.883584    5060 generic.go:356] "Generic (PLEG): container finished" podID="baf26f18-a9ed-4385-ae09-956af9788fec" containerID="f6d64137216977809d8b5a9ca44967343e3d692ce88a2a5ca754dcba4e9e1287" exitCode=0
Oct 30 12:46:01.883781 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:01.883672    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk" event={"ID":"baf26f18-a9ed-4385-ae09-956af9788fec","Type":"ContainerDied","Data":"f6d64137216977809d8b5a9ca44967343e3d692ce88a2a5ca754dcba4e9e1287"}
Oct 30 12:46:02.324612 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:02.316850    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:02.573360 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:02.573323    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:03.060503 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.060478    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:03.063677 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.063654    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:03.197565 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.197539    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca\") pod \"c6a84611-bc65-4e69-908c-8a04c92c629f\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") "
Oct 30 12:46:03.197769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.197588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca\") pod \"baf26f18-a9ed-4385-ae09-956af9788fec\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") "
Oct 30 12:46:03.197769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.197627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca\") pod \"c6a84611-bc65-4e69-908c-8a04c92c629f\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") "
Oct 30 12:46:03.197769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.197702    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca\") pod \"baf26f18-a9ed-4385-ae09-956af9788fec\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") "
Oct 30 12:46:03.197769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.197743    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4\" (UniqueName: \"kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4\") pod \"baf26f18-a9ed-4385-ae09-956af9788fec\" (UID: \"baf26f18-a9ed-4385-ae09-956af9788fec\") "
Oct 30 12:46:03.198030 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.197834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv\" (UniqueName: \"kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv\") pod \"c6a84611-bc65-4e69-908c-8a04c92c629f\" (UID: \"c6a84611-bc65-4e69-908c-8a04c92c629f\") "
Oct 30 12:46:03.198099 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.198076    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca" pod "c6a84611-bc65-4e69-908c-8a04c92c629f" (UID: "c6a84611-bc65-4e69-908c-8a04c92c629f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:03.198157 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.198102    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca" pod "baf26f18-a9ed-4385-ae09-956af9788fec" (UID: "baf26f18-a9ed-4385-ae09-956af9788fec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:03.198268 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.198248    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:03.198544 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.198271    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:03.200036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.200011    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca" pod "c6a84611-bc65-4e69-908c-8a04c92c629f" (UID: "c6a84611-bc65-4e69-908c-8a04c92c629f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:03.200036 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.200019    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4" pod "baf26f18-a9ed-4385-ae09-956af9788fec" (UID: "baf26f18-a9ed-4385-ae09-956af9788fec"). InnerVolumeSpecName "kube-api-access-h57q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:03.200522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.200501    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca" pod "baf26f18-a9ed-4385-ae09-956af9788fec" (UID: "baf26f18-a9ed-4385-ae09-956af9788fec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:03.200644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.200522    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv" pod "c6a84611-bc65-4e69-908c-8a04c92c629f" (UID: "c6a84611-bc65-4e69-908c-8a04c92c629f"). InnerVolumeSpecName "kube-api-access-cbsnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:03.299672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.299640    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsnv\" (UniqueName: \"kubernetes.io/projected/c6a84611-bc65-4e69-908c-8a04c92c629f-kube-api-access-cbsnv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:03.299672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.299666    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a84611-bc65-4e69-908c-8a04c92c629f-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:03.299672 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.299676    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf26f18-a9ed-4385-ae09-956af9788fec-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:03.299931 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.299685    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h57q4\" (UniqueName: \"kubernetes.io/projected/baf26f18-a9ed-4385-ae09-956af9788fec-kube-api-access-h57q4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:03.902827 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.902798    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n"
Oct 30 12:46:03.903027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.902788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0afeef5-kkk2n" event={"ID":"c6a84611-bc65-4e69-908c-8a04c92c629f","Type":"ContainerDied","Data":"1fa6331b6dd8616d8f55f324e7d8be9492625acaecdf0d665a87d2ff9b22c7e1"}
Oct 30 12:46:03.903027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.902973    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fa6331b6dd8616d8f55f324e7d8be9492625acaecdf0d665a87d2ff9b22c7e1"
Oct 30 12:46:03.904506 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.904479    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk" event={"ID":"baf26f18-a9ed-4385-ae09-956af9788fec","Type":"ContainerDied","Data":"a717f24e8daf970105e77652adc812cb73a813c896e0e3f6a41c185eb06e9b55"}
Oct 30 12:46:03.904662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.904526    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a717f24e8daf970105e77652adc812cb73a813c896e0e3f6a41c185eb06e9b55"
Oct 30 12:46:03.904662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:03.904484    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c329b-l48vk"
Oct 30 12:46:04.066674 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:04.066646    5060 scope.go:122] "RemoveContainer" containerID="7f6883062dcbef4ca0127a8a1ca5c290bfcd1108f9939274a5811ceb12456581"
Oct 30 12:46:04.078770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:04.078743    5060 scope.go:122] "RemoveContainer" containerID="77a61f344178bd04935a1cb2ba9a0254763348002289339a0f386c8cadd05de6"
Oct 30 12:46:04.090064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:04.090039    5060 scope.go:122] "RemoveContainer" containerID="1e4de927b23b735048970da3a6254556d125ef478739f245ad261bfd50c44dd9"
Oct 30 12:46:04.156341 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:04.156306    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:05.930249 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:05.930212    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:05.993468 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:05.993417    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"]
Oct 30 12:46:05.993720 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:05.993693    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy" containerID="cri-o://619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db" gracePeriod=30
Oct 30 12:46:07.106885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.106849    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.240656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.240619    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.370921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.370837    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"]
Oct 30 12:46:07.371126 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.371098    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy" containerID="cri-o://706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801" gracePeriod=30
Oct 30 12:46:07.402980 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.402904    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" containerID="cri-o://456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a" gracePeriod=45
Oct 30 12:46:07.441269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.441237    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.501201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.501159    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 30 12:46:07.669588 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.669556    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 12:46:07.748160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748112    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx\" (UniqueName: \"kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx\") pod \"0ed39ccb-e93a-40af-9214-e566b04e05f7\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") "
Oct 30 12:46:07.748160 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748158    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home\") pod \"0ed39ccb-e93a-40af-9214-e566b04e05f7\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") "
Oct 30 12:46:07.748539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748181    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache\") pod \"0ed39ccb-e93a-40af-9214-e566b04e05f7\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") "
Oct 30 12:46:07.748539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748216    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir\") pod \"0ed39ccb-e93a-40af-9214-e566b04e05f7\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") "
Oct 30 12:46:07.748539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748250    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs\") pod \"0ed39ccb-e93a-40af-9214-e566b04e05f7\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") "
Oct 30 12:46:07.748539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748375    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm\") pod \"0ed39ccb-e93a-40af-9214-e566b04e05f7\" (UID: \"0ed39ccb-e93a-40af-9214-e566b04e05f7\") "
Oct 30 12:46:07.748818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748600    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home" pod "0ed39ccb-e93a-40af-9214-e566b04e05f7" (UID: "0ed39ccb-e93a-40af-9214-e566b04e05f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.748818 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748743    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache" pod "0ed39ccb-e93a-40af-9214-e566b04e05f7" (UID: "0ed39ccb-e93a-40af-9214-e566b04e05f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.749018 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.748990    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.749174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.749021    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.749174 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.749030    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir" pod "0ed39ccb-e93a-40af-9214-e566b04e05f7" (UID: "0ed39ccb-e93a-40af-9214-e566b04e05f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.750941 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.750913    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm" pod "0ed39ccb-e93a-40af-9214-e566b04e05f7" (UID: "0ed39ccb-e93a-40af-9214-e566b04e05f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.751047 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.751004    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs" pod "0ed39ccb-e93a-40af-9214-e566b04e05f7" (UID: "0ed39ccb-e93a-40af-9214-e566b04e05f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:07.753960 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.753934    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx" pod "0ed39ccb-e93a-40af-9214-e566b04e05f7" (UID: "0ed39ccb-e93a-40af-9214-e566b04e05f7"). InnerVolumeSpecName "kube-api-access-2k2gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:07.757897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.757857    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.796557 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.796517    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main" containerID="cri-o://9828724cd2ea77d5d92b5681ecd7b2619da3519005ad16ffb9d0c6d16ddadb9f" gracePeriod=45
Oct 30 12:46:07.850133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.850096    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.850133 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.850133    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2gx\" (UniqueName: \"kubernetes.io/projected/0ed39ccb-e93a-40af-9214-e566b04e05f7-kube-api-access-2k2gx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.850361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.850148    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed39ccb-e93a-40af-9214-e566b04e05f7-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.850361 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.850161    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed39ccb-e93a-40af-9214-e566b04e05f7-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.918427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.918398    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.937690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.937616    5060 generic.go:356] "Generic (PLEG): container finished" podID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerID="9828724cd2ea77d5d92b5681ecd7b2619da3519005ad16ffb9d0c6d16ddadb9f" exitCode=0
Oct 30 12:46:07.937690 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.937672    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" event={"ID":"26fdd688-a64c-4ef0-95d5-3e515bdc171c","Type":"ContainerDied","Data":"9828724cd2ea77d5d92b5681ecd7b2619da3519005ad16ffb9d0c6d16ddadb9f"}
Oct 30 12:46:07.939485 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.939425    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerID="456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a" exitCode=0
Oct 30 12:46:07.939610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.939480    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" event={"ID":"0ed39ccb-e93a-40af-9214-e566b04e05f7","Type":"ContainerDied","Data":"456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a"}
Oct 30 12:46:07.939610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.939518    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"
Oct 30 12:46:07.939610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.939525    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4" event={"ID":"0ed39ccb-e93a-40af-9214-e566b04e05f7","Type":"ContainerDied","Data":"6d3611182054f4f7bc65e186212e28a6705210a40c4e0d8c08364721448156f0"}
Oct 30 12:46:07.939610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.939548    5060 scope.go:122] "RemoveContainer" containerID="456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a"
Oct 30 12:46:07.951644 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.951626    5060 scope.go:122] "RemoveContainer" containerID="456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a"
Oct 30 12:46:07.951948 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:07.951925    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a\": container with ID starting with 456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a not found: ID does not exist" containerID="456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a"
Oct 30 12:46:07.952021 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.951957    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a"} err="failed to get container status \"456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a\": rpc error: code = NotFound desc = could not find container \"456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a\": container with ID starting with 456a4009ea4767d57a18adaf3238b062953e975d78f0eefbe2a991f07404837a not found: ID does not exist"
Oct 30 12:46:07.995051 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.995022    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"]
Oct 30 12:46:08.000012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:07.999982    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/simulator-w2-1d9b1a-kserve-667578549c-598p4"]
Oct 30 12:46:08.049357 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.049331    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 12:46:08.153140 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153110    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir\") pod \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") "
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153210    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm\") pod \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") "
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153237    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home\") pod \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") "
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153264    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs\" (UniqueName: \"kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs\") pod \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") "
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153284    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x\" (UniqueName: \"kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x\") pod \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") "
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153323    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache\") pod \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\" (UID: \"26fdd688-a64c-4ef0-95d5-3e515bdc171c\") "
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153506    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir" pod "26fdd688-a64c-4ef0-95d5-3e515bdc171c" (UID: "26fdd688-a64c-4ef0-95d5-3e515bdc171c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.153662 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153588    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home" pod "26fdd688-a64c-4ef0-95d5-3e515bdc171c" (UID: "26fdd688-a64c-4ef0-95d5-3e515bdc171c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.154335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153772    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.154335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache" pod "26fdd688-a64c-4ef0-95d5-3e515bdc171c" (UID: "26fdd688-a64c-4ef0-95d5-3e515bdc171c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.154335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.153791    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.155656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.155632    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x" pod "26fdd688-a64c-4ef0-95d5-3e515bdc171c" (UID: "26fdd688-a64c-4ef0-95d5-3e515bdc171c"). InnerVolumeSpecName "kube-api-access-7lz5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:08.155656 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.155636    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm" pod "26fdd688-a64c-4ef0-95d5-3e515bdc171c" (UID: "26fdd688-a64c-4ef0-95d5-3e515bdc171c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.155803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.155663    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs" pod "26fdd688-a64c-4ef0-95d5-3e515bdc171c" (UID: "26fdd688-a64c-4ef0-95d5-3e515bdc171c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:08.255333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.255296    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.255333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.255332    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fdd688-a64c-4ef0-95d5-3e515bdc171c-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.255614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.255346    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lz5x\" (UniqueName: \"kubernetes.io/projected/26fdd688-a64c-4ef0-95d5-3e515bdc171c-kube-api-access-7lz5x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.255614 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.255358    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fdd688-a64c-4ef0-95d5-3e515bdc171c-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.292751 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.292696    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main" containerID="cri-o://87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32" gracePeriod=45
Oct 30 12:46:08.309898 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.309862    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main" containerID="cri-o://494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07" gracePeriod=45
Oct 30 12:46:08.333691 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.333661    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" path="/var/lib/kubelet/pods/0ed39ccb-e93a-40af-9214-e566b04e05f7/volumes"
Oct 30 12:46:08.574623 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.574596    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 12:46:08.611647 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.611615    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 12:46:08.659738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.659362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz\" (UniqueName: \"kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz\") pod \"e52f4f19-a1b7-4948-b692-8560d0d8b846\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") "
Oct 30 12:46:08.659738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.659418    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache\") pod \"e52f4f19-a1b7-4948-b692-8560d0d8b846\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") "
Oct 30 12:46:08.659738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.659514    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm\") pod \"e52f4f19-a1b7-4948-b692-8560d0d8b846\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") "
Oct 30 12:46:08.659738 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.659588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs\") pod \"e52f4f19-a1b7-4948-b692-8560d0d8b846\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") "
Oct 30 12:46:08.660384 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.660361    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home\") pod \"e52f4f19-a1b7-4948-b692-8560d0d8b846\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") "
Oct 30 12:46:08.660538 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.660425    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir\") pod \"e52f4f19-a1b7-4948-b692-8560d0d8b846\" (UID: \"e52f4f19-a1b7-4948-b692-8560d0d8b846\") "
Oct 30 12:46:08.660683 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.660658    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache" pod "e52f4f19-a1b7-4948-b692-8560d0d8b846" (UID: "e52f4f19-a1b7-4948-b692-8560d0d8b846"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.660904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.660856    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir" pod "e52f4f19-a1b7-4948-b692-8560d0d8b846" (UID: "e52f4f19-a1b7-4948-b692-8560d0d8b846"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.660904 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.660892    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.661860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.661824    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home" pod "e52f4f19-a1b7-4948-b692-8560d0d8b846" (UID: "e52f4f19-a1b7-4948-b692-8560d0d8b846"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.664854 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.664531    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm" pod "e52f4f19-a1b7-4948-b692-8560d0d8b846" (UID: "e52f4f19-a1b7-4948-b692-8560d0d8b846"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.665115 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.665092    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz" pod "e52f4f19-a1b7-4948-b692-8560d0d8b846" (UID: "e52f4f19-a1b7-4948-b692-8560d0d8b846"). InnerVolumeSpecName "kube-api-access-86xwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:08.665216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.665153    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs" pod "e52f4f19-a1b7-4948-b692-8560d0d8b846" (UID: "e52f4f19-a1b7-4948-b692-8560d0d8b846"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:08.761474 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761429    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache\") pod \"fe201112-afab-4c2c-bb35-b1ea90fc281b\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") "
Oct 30 12:46:08.761676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs\") pod \"fe201112-afab-4c2c-bb35-b1ea90fc281b\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") "
Oct 30 12:46:08.761676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761621    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home\") pod \"fe201112-afab-4c2c-bb35-b1ea90fc281b\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") "
Oct 30 12:46:08.761821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761687    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir\") pod \"fe201112-afab-4c2c-bb35-b1ea90fc281b\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") "
Oct 30 12:46:08.761821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761718    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm\") pod \"fe201112-afab-4c2c-bb35-b1ea90fc281b\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") "
Oct 30 12:46:08.761821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache" pod "fe201112-afab-4c2c-bb35-b1ea90fc281b" (UID: "fe201112-afab-4c2c-bb35-b1ea90fc281b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.761821 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761745    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g\" (UniqueName: \"kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g\") pod \"fe201112-afab-4c2c-bb35-b1ea90fc281b\" (UID: \"fe201112-afab-4c2c-bb35-b1ea90fc281b\") "
Oct 30 12:46:08.762049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761955    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home" pod "fe201112-afab-4c2c-bb35-b1ea90fc281b" (UID: "fe201112-afab-4c2c-bb35-b1ea90fc281b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.762049 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.761990    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir" pod "fe201112-afab-4c2c-bb35-b1ea90fc281b" (UID: "fe201112-afab-4c2c-bb35-b1ea90fc281b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.762193 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762176    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762198    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762210    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762222    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762235    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86xwz\" (UniqueName: \"kubernetes.io/projected/e52f4f19-a1b7-4948-b692-8560d0d8b846-kube-api-access-86xwz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762252 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762248    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4f19-a1b7-4948-b692-8560d0d8b846-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762262    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.762515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.762275    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4f19-a1b7-4948-b692-8560d0d8b846-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.764190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.764162    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs" pod "fe201112-afab-4c2c-bb35-b1ea90fc281b" (UID: "fe201112-afab-4c2c-bb35-b1ea90fc281b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:08.764311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.764226    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm" pod "fe201112-afab-4c2c-bb35-b1ea90fc281b" (UID: "fe201112-afab-4c2c-bb35-b1ea90fc281b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.764311 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.764263    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g" pod "fe201112-afab-4c2c-bb35-b1ea90fc281b" (UID: "fe201112-afab-4c2c-bb35-b1ea90fc281b"). InnerVolumeSpecName "kube-api-access-vps4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:08.863278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.863152    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe201112-afab-4c2c-bb35-b1ea90fc281b-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.863278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.863184    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe201112-afab-4c2c-bb35-b1ea90fc281b-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.863278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.863194    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vps4g\" (UniqueName: \"kubernetes.io/projected/fe201112-afab-4c2c-bb35-b1ea90fc281b-kube-api-access-vps4g\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.947810 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.947772    5060 generic.go:356] "Generic (PLEG): container finished" podID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerID="494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07" exitCode=0
Oct 30 12:46:08.947965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.947817    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"
Oct 30 12:46:08.947965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.947850    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" event={"ID":"fe201112-afab-4c2c-bb35-b1ea90fc281b","Type":"ContainerDied","Data":"494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07"}
Oct 30 12:46:08.947965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.947892    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp" event={"ID":"fe201112-afab-4c2c-bb35-b1ea90fc281b","Type":"ContainerDied","Data":"9e67b5dc4fa040a5acf6af1d977e8ad62c0086ca0d5afc0378215006b83762bc"}
Oct 30 12:46:08.947965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.947915    5060 scope.go:122] "RemoveContainer" containerID="494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07"
Oct 30 12:46:08.949541 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.949514    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"
Oct 30 12:46:08.949676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.949561    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54" event={"ID":"26fdd688-a64c-4ef0-95d5-3e515bdc171c","Type":"ContainerDied","Data":"fdbb7efab8b1dd033e052ba4e478b46646dd2bf3476c6e6ae22ca2988f70fba7"}
Oct 30 12:46:08.952509 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.952487    5060 generic.go:356] "Generic (PLEG): container finished" podID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerID="87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32" exitCode=0
Oct 30 12:46:08.952629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.952521    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"
Oct 30 12:46:08.952629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.952529    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" event={"ID":"e52f4f19-a1b7-4948-b692-8560d0d8b846","Type":"ContainerDied","Data":"87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32"}
Oct 30 12:46:08.952629 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.952559    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp" event={"ID":"e52f4f19-a1b7-4948-b692-8560d0d8b846","Type":"ContainerDied","Data":"bbfe1a08dcaef4a8c2f63e6d3fe4bb5b48559e40c8739c0cb7ad96d66b4a13e7"}
Oct 30 12:46:08.960015 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.959993    5060 scope.go:122] "RemoveContainer" containerID="494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07"
Oct 30 12:46:08.960270 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:08.960252    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07\": container with ID starting with 494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07 not found: ID does not exist" containerID="494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07"
Oct 30 12:46:08.960340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.960278    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07"} err="failed to get container status \"494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07\": rpc error: code = NotFound desc = could not find container \"494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07\": container with ID starting with 494138f071475abef37c74efaed696c1086239a89df499bd5a002723416f4e07 not found: ID does not exist"
Oct 30 12:46:08.960340 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.960295    5060 scope.go:122] "RemoveContainer" containerID="9828724cd2ea77d5d92b5681ecd7b2619da3519005ad16ffb9d0c6d16ddadb9f"
Oct 30 12:46:08.970012 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.969995    5060 scope.go:122] "RemoveContainer" containerID="87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32"
Oct 30 12:46:08.978843 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.978828    5060 scope.go:122] "RemoveContainer" containerID="87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32"
Oct 30 12:46:08.979127 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:08.979099    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32\": container with ID starting with 87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32 not found: ID does not exist" containerID="87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32"
Oct 30 12:46:08.979205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:08.979136    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32"} err="failed to get container status \"87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32\": rpc error: code = NotFound desc = could not find container \"87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32\": container with ID starting with 87c0f06d4ea54346707469cbb94a67ad0f37a163ced1c3666c7b37dc5dd3df32 not found: ID does not exist"
Oct 30 12:46:09.003973 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.003950    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"]
Oct 30 12:46:09.012369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.012348    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/premium-w2-1d9b1a-kserve-7584d988d8-tvg54"]
Oct 30 12:46:09.045663 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.045642    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"]
Oct 30 12:46:09.051817 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.051794    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-w0-3ccdab-kserve-5f67484c47-bnchp"]
Oct 30 12:46:09.091659 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.091623    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"]
Oct 30 12:46:09.099637 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.099613    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/simulator-w0-3ccdab-kserve-6c6c8ddb9-8lssp"]
Oct 30 12:46:09.906789 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:09.906748    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:10.333885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:10.333853    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" path="/var/lib/kubelet/pods/26fdd688-a64c-4ef0-95d5-3e515bdc171c/volumes"
Oct 30 12:46:10.334266 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:10.334252    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" path="/var/lib/kubelet/pods/e52f4f19-a1b7-4948-b692-8560d0d8b846/volumes"
Oct 30 12:46:10.334634 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:10.334621    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" path="/var/lib/kubelet/pods/fe201112-afab-4c2c-bb35-b1ea90fc281b/volumes"
Oct 30 12:46:11.245519 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.245496    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:46:11.391989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.391916    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.391989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.391957    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392008    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392055    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392216 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392175    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392225    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392260    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392290    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy\") pod \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\" (UID: \"6f3b99f3-488c-41a0-a86b-9615f6d6aca7\") "
Oct 30 12:46:11.392501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392335    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:11.392501 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392393    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392506    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392524    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392656    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392674    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392695    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392710    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.392778 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.392723    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.394773 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.394750    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:11.394887 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.394779    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:11.395062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.395042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "kube-api-access-wlh5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:11.395104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.395065    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token" pod "6f3b99f3-488c-41a0-a86b-9615f6d6aca7" (UID: "6f3b99f3-488c-41a0-a86b-9615f6d6aca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:11.493958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.493930    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.493958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.493957    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.494153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.493967    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.494153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.493975    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.494153 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.493984    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlh5q\" (UniqueName: \"kubernetes.io/projected/6f3b99f3-488c-41a0-a86b-9615f6d6aca7-kube-api-access-wlh5q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:11.758725 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.758677    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:11.980239 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.980203    5060 generic.go:356] "Generic (PLEG): container finished" podID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerID="619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db" exitCode=0
Oct 30 12:46:11.980394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.980262    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"
Oct 30 12:46:11.980394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.980294    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" event={"ID":"6f3b99f3-488c-41a0-a86b-9615f6d6aca7","Type":"ContainerDied","Data":"619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db"}
Oct 30 12:46:11.980394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.980359    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl" event={"ID":"6f3b99f3-488c-41a0-a86b-9615f6d6aca7","Type":"ContainerDied","Data":"3c395f582cfcc26fefceb960cef51abfc13ecdcb5f5bc2df6f8ec632d60a165b"}
Oct 30 12:46:11.980394 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.980391    5060 scope.go:122] "RemoveContainer" containerID="619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db"
Oct 30 12:46:11.991924 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.991905    5060 scope.go:122] "RemoveContainer" containerID="619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db"
Oct 30 12:46:11.992190 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:11.992172    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db\": container with ID starting with 619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db not found: ID does not exist" containerID="619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db"
Oct 30 12:46:11.992234 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:11.992199    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db"} err="failed to get container status \"619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db\": rpc error: code = NotFound desc = could not find container \"619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db\": container with ID starting with 619f49eebd71cf9f5a35eab8e41d9c66cd8deaaaa2efe80b72edf687cb9967db not found: ID does not exist"
Oct 30 12:46:12.046676 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.046620    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"]
Oct 30 12:46:12.050284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.050263    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fwdcl"]
Oct 30 12:46:12.333900 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.333818    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" path="/var/lib/kubelet/pods/6f3b99f3-488c-41a0-a86b-9615f6d6aca7/volumes"
Oct 30 12:46:12.575733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.575693    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:12.645956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.645929    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:46:12.804362 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804326    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804372    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804401    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804482    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804611 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804540    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804613    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804672    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804714    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket\") pod \"892354bd-b329-4d9f-8b3e-a2eda63c55af\" (UID: \"892354bd-b329-4d9f-8b3e-a2eda63c55af\") "
Oct 30 12:46:12.804977 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.804759    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:12.805201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.805006    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:12.805201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.805020    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.805201 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.805031    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:12.805370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.805198    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:12.805370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.805312    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:12.807108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.807080    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:12.807314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.807291    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:12.807314 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.807300    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:12.807441 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.807328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj" pod "892354bd-b329-4d9f-8b3e-a2eda63c55af" (UID: "892354bd-b329-4d9f-8b3e-a2eda63c55af"). InnerVolumeSpecName "kube-api-access-gs5wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:12.849581 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.849552    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"]
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906402    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906433    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906447    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906475    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5wj\" (UniqueName: \"kubernetes.io/projected/892354bd-b329-4d9f-8b3e-a2eda63c55af-kube-api-access-gs5wj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906489    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892354bd-b329-4d9f-8b3e-a2eda63c55af-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906501    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892354bd-b329-4d9f-8b3e-a2eda63c55af-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906512    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.906527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.906524    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892354bd-b329-4d9f-8b3e-a2eda63c55af-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.991872 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.991835    5060 generic.go:356] "Generic (PLEG): container finished" podID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerID="706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801" exitCode=0
Oct 30 12:46:12.992053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.991896    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"
Oct 30 12:46:12.992053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.991916    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" event={"ID":"892354bd-b329-4d9f-8b3e-a2eda63c55af","Type":"ContainerDied","Data":"706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801"}
Oct 30 12:46:12.992053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.991969    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2" event={"ID":"892354bd-b329-4d9f-8b3e-a2eda63c55af","Type":"ContainerDied","Data":"979f95adb896967391e2e13a0d20de1fd014a606fd4ca3167d2dde40b8c392a9"}
Oct 30 12:46:12.992053 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.991984    5060 scope.go:122] "RemoveContainer" containerID="706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801"
Oct 30 12:46:12.993529 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:12.993390    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:13.006267 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.006251    5060 scope.go:122] "RemoveContainer" containerID="3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"
Oct 30 12:46:13.016240 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.016222    5060 scope.go:122] "RemoveContainer" containerID="706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801"
Oct 30 12:46:13.016533 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:13.016506    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801\": container with ID starting with 706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801 not found: ID does not exist" containerID="706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801"
Oct 30 12:46:13.016583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.016545    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801"} err="failed to get container status \"706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801\": rpc error: code = NotFound desc = could not find container \"706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801\": container with ID starting with 706a485e0e3c8d8f529bdeb72ba1364f06a8baf173c1c4ff4e0b34a3b71f9801 not found: ID does not exist"
Oct 30 12:46:13.016583 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.016567    5060 scope.go:122] "RemoveContainer" containerID="3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"
Oct 30 12:46:13.016801 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:13.016785    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9\": container with ID starting with 3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9 not found: ID does not exist" containerID="3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"
Oct 30 12:46:13.016844 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.016806    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9"} err="failed to get container status \"3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9\": rpc error: code = NotFound desc = could not find container \"3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9\": container with ID starting with 3d7f1b1b428f0198f3b1f5dd3e3295fae859ed11c7795e829367b19fcde7d4a9 not found: ID does not exist"
Oct 30 12:46:13.045908 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.045882    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"]
Oct 30 12:46:13.051705 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.051681    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bc4e2-openshift-default-75d485bdf-5p5h2"]
Oct 30 12:46:13.304237 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.304198    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:13.695358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.695267    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:13.846335 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:13.846300    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:14.108108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.108072    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:14.335613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.335578    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" path="/var/lib/kubelet/pods/892354bd-b329-4d9f-8b3e-a2eda63c55af/volumes"
Oct 30 12:46:14.362035 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.361956    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:14.474868 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.474830    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"]
Oct 30 12:46:14.480965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.480938    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1d9b1a-d2msp"]
Oct 30 12:46:14.532850 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.532819    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:14.972444 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:14.972412    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:16.333989 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:16.333950    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56" path="/var/lib/kubelet/pods/c3fc9b59-dd58-493a-ae7a-96ef7fcd8e56/volumes"
Oct 30 12:46:16.344149 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:16.344120    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:17.001064 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.001024    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:17.143967 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.143934    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:17.497230 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.497199    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:46:17.652176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.652083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh\" (UniqueName: \"kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh\") pod \"538d1600-10e2-451c-9fdb-67beaaa2335d\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") "
Oct 30 12:46:17.652176 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.652118    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume\") pod \"538d1600-10e2-451c-9fdb-67beaaa2335d\" (UID: \"538d1600-10e2-451c-9fdb-67beaaa2335d\") "
Oct 30 12:46:17.652561 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.652537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume" pod "538d1600-10e2-451c-9fdb-67beaaa2335d" (UID: "538d1600-10e2-451c-9fdb-67beaaa2335d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:17.654427 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.654405    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh" pod "538d1600-10e2-451c-9fdb-67beaaa2335d" (UID: "538d1600-10e2-451c-9fdb-67beaaa2335d"). InnerVolumeSpecName "kube-api-access-86mjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:17.753783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.753743    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mjh\" (UniqueName: \"kubernetes.io/projected/538d1600-10e2-451c-9fdb-67beaaa2335d-kube-api-access-86mjh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:17.753783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:17.753776    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538d1600-10e2-451c-9fdb-67beaaa2335d-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:18.047576 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.047541    5060 generic.go:356] "Generic (PLEG): container finished" podID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerID="40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a" exitCode=0
Oct 30 12:46:18.047800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.047585    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"
Oct 30 12:46:18.047800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.047627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" event={"ID":"538d1600-10e2-451c-9fdb-67beaaa2335d","Type":"ContainerDied","Data":"40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a"}
Oct 30 12:46:18.047800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.047685    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx" event={"ID":"538d1600-10e2-451c-9fdb-67beaaa2335d","Type":"ContainerDied","Data":"c276cf86c971064478eaaad1916262220f379c4fd8bebd0fd515250b1a64bedd"}
Oct 30 12:46:18.047800 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.047714    5060 scope.go:122] "RemoveContainer" containerID="40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a"
Oct 30 12:46:18.058434 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.058412    5060 scope.go:122] "RemoveContainer" containerID="40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a"
Oct 30 12:46:18.058719 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:18.058701    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a\": container with ID starting with 40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a not found: ID does not exist" containerID="40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a"
Oct 30 12:46:18.058783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.058728    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a"} err="failed to get container status \"40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a\": rpc error: code = NotFound desc = could not find container \"40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a\": container with ID starting with 40d5f722a330179b3935776dc62c57c9c3b12f8230da20bf13188d388328857a not found: ID does not exist"
Oct 30 12:46:18.101471 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.101422    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"]
Oct 30 12:46:18.112608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.112577    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bc4e2-5f48589cf4-9mhnx"]
Oct 30 12:46:18.333733 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:18.333649    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" path="/var/lib/kubelet/pods/538d1600-10e2-451c-9fdb-67beaaa2335d/volumes"
Oct 30 12:46:19.503522 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503486    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"]
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503878    5060 cpu_manager.go:414] "containerMap: removing container" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503888    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503898    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503903    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503914    5060 cpu_manager.go:414] "containerMap: removing container" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503919    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503935    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main"
Oct 30 12:46:19.503942 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503941    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503950    5060 cpu_manager.go:414] "containerMap: removing container" podUID="baf26f18-a9ed-4385-ae09-956af9788fec" containerName="revoke-keys"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503955    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf26f18-a9ed-4385-ae09-956af9788fec" containerName="revoke-keys"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503966    5060 cpu_manager.go:414] "containerMap: removing container" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerName="payload-processing"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503970    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerName="payload-processing"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503977    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503981    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503989    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c6a84611-bc65-4e69-908c-8a04c92c629f" containerName="revoke-keys"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.503993    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a84611-bc65-4e69-908c-8a04c92c629f" containerName="revoke-keys"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504007    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504011    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504065    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504073    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fdd688-a64c-4ef0-95d5-3e515bdc171c" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504079    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe201112-afab-4c2c-bb35-b1ea90fc281b" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504086    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504093    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed39ccb-e93a-40af-9214-e566b04e05f7" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504097    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf26f18-a9ed-4385-ae09-956af9788fec" containerName="revoke-keys"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504103    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="538d1600-10e2-451c-9fdb-67beaaa2335d" containerName="payload-processing"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504107    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52f4f19-a1b7-4948-b692-8560d0d8b846" containerName="main"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504113    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a84611-bc65-4e69-908c-8a04c92c629f" containerName="revoke-keys"
Oct 30 12:46:19.504208 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.504119    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3b99f3-488c-41a0-a86b-9615f6d6aca7" containerName="istio-proxy"
Oct 30 12:46:19.508909 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.508884    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.525498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.525474    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"]
Oct 30 12:46:19.534024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.534003    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:46:19.534186 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.534026    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:46:19.609274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.609235    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:46:19.609274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.609271    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:46:19.670756 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.670718    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.670927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.670763    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcz5\" (UniqueName: \"kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.670927 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.670882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.772000 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.771924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.772168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.772011    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.772168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.772053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcz5\" (UniqueName: \"kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.772722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.772695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.774463 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.774429    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.795396 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.795375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcz5\" (UniqueName: \"kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5\") pod \"maas-api-revoke-sub-e737eb13-m4lhw\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.825846 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.825824    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:19.857666 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:19.857625    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:46:19.985698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:19.985661    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:20.127679 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:20.127599    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:20.131062 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:20.131033    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"]
Oct 30 12:46:20.131207 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:46:20.131185    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330a62e_dc32_48cb_8ebd_2afafbf511df.slice/crio-e15e7840b67642a653ec3b325efe8de77b0f04b6b95000964c12ae6429af5140 WatchSource:0}: Error finding container e15e7840b67642a653ec3b325efe8de77b0f04b6b95000964c12ae6429af5140: Status 404 returned error can't find the container with id e15e7840b67642a653ec3b325efe8de77b0f04b6b95000964c12ae6429af5140
Oct 30 12:46:21.076932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:21.076895    5060 generic.go:356] "Generic (PLEG): container finished" podID="a330a62e-dc32-48cb-8ebd-2afafbf511df" containerID="4bc7556b30ef096f89d10e7fd544576de1fe3cf2d37e2d98cad5aaa1d8987a55" exitCode=0
Oct 30 12:46:21.077365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:21.076980    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw" event={"ID":"a330a62e-dc32-48cb-8ebd-2afafbf511df","Type":"ContainerDied","Data":"4bc7556b30ef096f89d10e7fd544576de1fe3cf2d37e2d98cad5aaa1d8987a55"}
Oct 30 12:46:21.077365 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:21.077035    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw" event={"ID":"a330a62e-dc32-48cb-8ebd-2afafbf511df","Type":"ContainerStarted","Data":"e15e7840b67642a653ec3b325efe8de77b0f04b6b95000964c12ae6429af5140"}
Oct 30 12:46:22.030985 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.030936    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:22.175308 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.175261    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:22.259998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.259964    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:22.397856 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.397577    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:22.398996 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.398966    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca\") pod \"a330a62e-dc32-48cb-8ebd-2afafbf511df\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") "
Oct 30 12:46:22.399173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.399055    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca\") pod \"a330a62e-dc32-48cb-8ebd-2afafbf511df\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") "
Oct 30 12:46:22.399173 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.399113    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5\" (UniqueName: \"kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5\") pod \"a330a62e-dc32-48cb-8ebd-2afafbf511df\" (UID: \"a330a62e-dc32-48cb-8ebd-2afafbf511df\") "
Oct 30 12:46:22.400044 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.400004    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca" pod "a330a62e-dc32-48cb-8ebd-2afafbf511df" (UID: "a330a62e-dc32-48cb-8ebd-2afafbf511df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:22.404435 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.404405    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5" pod "a330a62e-dc32-48cb-8ebd-2afafbf511df" (UID: "a330a62e-dc32-48cb-8ebd-2afafbf511df"). InnerVolumeSpecName "kube-api-access-2vcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:22.411666 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.411070    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca" pod "a330a62e-dc32-48cb-8ebd-2afafbf511df" (UID: "a330a62e-dc32-48cb-8ebd-2afafbf511df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:22.509475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.506104    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-openshift-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:22.509475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.506144    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vcz5\" (UniqueName: \"kubernetes.io/projected/a330a62e-dc32-48cb-8ebd-2afafbf511df-kube-api-access-2vcz5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:22.509475 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.506161    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330a62e-dc32-48cb-8ebd-2afafbf511df-maas-api-xks-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:22.536108 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:22.536064    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:23.095498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:23.093967    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw"
Oct 30 12:46:23.095498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:23.093998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e737eb13-m4lhw" event={"ID":"a330a62e-dc32-48cb-8ebd-2afafbf511df","Type":"ContainerDied","Data":"e15e7840b67642a653ec3b325efe8de77b0f04b6b95000964c12ae6429af5140"}
Oct 30 12:46:23.095498 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:23.094059    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e15e7840b67642a653ec3b325efe8de77b0f04b6b95000964c12ae6429af5140"
Oct 30 12:46:27.870188 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:27.870134    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main" containerID="cri-o://693cb18ce1d04e679209ac7a5b7eba2fc069baea68d8e165a49fd1bc605647db" gracePeriod=45
Oct 30 12:46:28.133805 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.133738    5060 generic.go:356] "Generic (PLEG): container finished" podID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerID="693cb18ce1d04e679209ac7a5b7eba2fc069baea68d8e165a49fd1bc605647db" exitCode=0
Oct 30 12:46:28.133965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.133810    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" event={"ID":"e32f971d-ffd4-4f01-b040-a6ff204125c5","Type":"ContainerDied","Data":"693cb18ce1d04e679209ac7a5b7eba2fc069baea68d8e165a49fd1bc605647db"}
Oct 30 12:46:28.133965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.133848    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq" event={"ID":"e32f971d-ffd4-4f01-b040-a6ff204125c5","Type":"ContainerDied","Data":"88541a75d811abb2311d596f99a093b83bae60328f730237bd56cecc967d9eac"}
Oct 30 12:46:28.133965 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.133874    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88541a75d811abb2311d596f99a093b83bae60328f730237bd56cecc967d9eac"
Oct 30 12:46:28.135113 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.135098    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 12:46:28.156089 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc\" (UniqueName: \"kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc\") pod \"e32f971d-ffd4-4f01-b040-a6ff204125c5\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") "
Oct 30 12:46:28.156248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156127    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache\") pod \"e32f971d-ffd4-4f01-b040-a6ff204125c5\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") "
Oct 30 12:46:28.156248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home\") pod \"e32f971d-ffd4-4f01-b040-a6ff204125c5\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") "
Oct 30 12:46:28.156248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156192    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs\") pod \"e32f971d-ffd4-4f01-b040-a6ff204125c5\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") "
Oct 30 12:46:28.156248 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156234    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir\") pod \"e32f971d-ffd4-4f01-b040-a6ff204125c5\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") "
Oct 30 12:46:28.156510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156259    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm\") pod \"e32f971d-ffd4-4f01-b040-a6ff204125c5\" (UID: \"e32f971d-ffd4-4f01-b040-a6ff204125c5\") "
Oct 30 12:46:28.156510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156382    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache" pod "e32f971d-ffd4-4f01-b040-a6ff204125c5" (UID: "e32f971d-ffd4-4f01-b040-a6ff204125c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.156510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156422    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home" pod "e32f971d-ffd4-4f01-b040-a6ff204125c5" (UID: "e32f971d-ffd4-4f01-b040-a6ff204125c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.156510 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156483    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir" pod "e32f971d-ffd4-4f01-b040-a6ff204125c5" (UID: "e32f971d-ffd4-4f01-b040-a6ff204125c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.156732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156678    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.156732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156698    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.156732 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.156712    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.158832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.158803    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs" pod "e32f971d-ffd4-4f01-b040-a6ff204125c5" (UID: "e32f971d-ffd4-4f01-b040-a6ff204125c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:28.158832 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.158817    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc" pod "e32f971d-ffd4-4f01-b040-a6ff204125c5" (UID: "e32f971d-ffd4-4f01-b040-a6ff204125c5"). InnerVolumeSpecName "kube-api-access-p6hjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:28.159269 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.159173    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm" pod "e32f971d-ffd4-4f01-b040-a6ff204125c5" (UID: "e32f971d-ffd4-4f01-b040-a6ff204125c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.257760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.257711    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32f971d-ffd4-4f01-b040-a6ff204125c5-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.257760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.257756    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32f971d-ffd4-4f01-b040-a6ff204125c5-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.257998 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:28.257770    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hjc\" (UniqueName: \"kubernetes.io/projected/e32f971d-ffd4-4f01-b040-a6ff204125c5-kube-api-access-p6hjc\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Oct 30 12:46:29.141739 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:29.141701    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"
Oct 30 12:46:29.196172 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:29.196135    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"]
Oct 30 12:46:29.206813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:29.206773    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-w1-e43a66-kserve-575d766f7-wltgq"]
Oct 30 12:46:29.989625 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:29.989582    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:30.137597 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:30.137562    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:30.333964 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:30.333889    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" path="/var/lib/kubelet/pods/e32f971d-ffd4-4f01-b040-a6ff204125c5/volumes"
Oct 30 12:46:31.310916 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:31.310881    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:31.862104 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:31.862066    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:32.003958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:32.003920    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:32.195608 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:32.195550    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:32.604057 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:32.603962    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:34.692024 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:34.691972    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:35.183146 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:35.183109    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:35.319469 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:35.319389    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:35.818264 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:35.818211    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:38.025812 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:38.025776    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:38.647836 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:38.647799    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:42.036382 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:42.036342    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:42.180358 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:42.180327    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:42.504068 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:42.504009    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:43.683770 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:43.683731    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-h4nph_c85b9afe-dc24-4cf9-8639-a59f727c4bb2/manager/1.log"
Oct 30 12:46:43.874564 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:43.874529    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lfxxm_d621ee63-03be-454b-b1b1-9e54a635433b/manager/0.log"
Oct 30 12:46:44.431205 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:44.431173    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7h64q_e1312742-65e5-4fd2-9751-0a260930048f/manager/0.log"
Oct 30 12:46:44.846512 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:44.846485    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pzz7z_ba17fa44-3f35-497d-b2ce-b4a6415511dc/manager/0.log"
Oct 30 12:46:45.284615 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:45.284584    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-h4nph_c85b9afe-dc24-4cf9-8639-a59f727c4bb2/manager/0.log"
Oct 30 12:46:46.050664 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:46.050638    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:46:46.249826 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:46.249794    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:46:46.464042 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:46.464016    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:46:47.133107 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:47.133077    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:46:47.561190 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:47.561161    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:46:49.091290 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:49.091257    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.232613 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:49.232569    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.785567 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:49.785518    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.858150 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:46:49.858112    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:46:51.210100 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:51.210068    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:56.999202 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:56.999165    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:57.469930 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:57.469900    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:59.283645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:59.283619    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:46:59.489552 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:59.489523    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:46:59.713243 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:46:59.713191    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:00.375027 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:00.375003    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:00.811642 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:00.811612    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:01.995761 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:01.995731    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-18c1f920-2cq49_4b21c73f-3c6f-498f-a0d5-2dc8ff383d7e/revoke-keys/0.log"
Oct 30 12:47:02.188859 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:02.188820    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-28e0340e-nlkxs_54f8fe96-6db6-455c-89f9-5beba9369c34/revoke-keys/0.log"
Oct 30 12:47:02.375416 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:02.375346    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4b7c329b-l48vk_baf26f18-a9ed-4385-ae09-956af9788fec/revoke-keys/0.log"
Oct 30 12:47:02.552142 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:02.552112    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8e35ae98-86wg4_40b9e9b9-53d0-4945-bdca-df9bcfd8e38e/revoke-keys/0.log"
Oct 30 12:47:02.742135 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:02.742106    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8ee3a86b-ckdlx_6e86adc1-ff0a-4daa-aa66-fece36e33add/revoke-keys/0.log"
Oct 30 12:47:02.934566 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:02.934531    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a2d7d9c6-45nq8_061ffc44-8b4b-4598-b062-a95fe670f510/revoke-keys/0.log"
Oct 30 12:47:03.127168 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:03.127095    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a304d978-jx7p2_949e431c-70d8-4c9d-aa44-2def91b2ef2f/revoke-keys/0.log"
Oct 30 12:47:03.324336 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:03.324307    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b0afeef5-kkk2n_c6a84611-bc65-4e69-908c-8a04c92c629f/revoke-keys/0.log"
Oct 30 12:47:03.690063 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:03.690031    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ce40f8a2-tbd45_af27c36f-c07a-43b9-bafe-d9b09d062792/revoke-keys/0.log"
Oct 30 12:47:04.069019 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.068983    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e737eb13-m4lhw_a330a62e-dc32-48cb-8ebd-2afafbf511df/revoke-keys/0.log"
Oct 30 12:47:04.238876 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.238845    5060 scope.go:122] "RemoveContainer" containerID="78ff77129b1632b91d3afaef6111663a28f4ed6d26ad9b3d6f8ab87339b88454"
Oct 30 12:47:04.249473 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.249440    5060 scope.go:122] "RemoveContainer" containerID="be352c81d725beaf20e15fac11c96fe6d26aa9c1a05c8211e66d74195c5f40b9"
Oct 30 12:47:04.261381 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.261361    5060 scope.go:122] "RemoveContainer" containerID="9df12acfc13c5e5cd7c4d597cf0e9201a0032198da7ec07fa4686a90c2fbe3df"
Oct 30 12:47:04.267878 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.267858    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f55322a1-f7n7f_8c276f3f-baf0-4bc2-b548-0d48554882c9/revoke-keys/0.log"
Oct 30 12:47:04.270884 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.270862    5060 scope.go:122] "RemoveContainer" containerID="1cb5892259d0ea1ded4bee83c99e6041ea145da295625b35b1a2207e3f3e6efa"
Oct 30 12:47:04.279700 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.279680    5060 scope.go:122] "RemoveContainer" containerID="7f294e93722e3f889a1a4bd3bb2eb18a3ec4895d9fa2962ac77ce8dc2eb67496"
Oct 30 12:47:04.288963 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.288942    5060 scope.go:122] "RemoveContainer" containerID="4ff70b8b6da82262701b439c2afc2480662ac68770a7c0a9459264d7cb3bf4d5"
Oct 30 12:47:04.297551 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.297530    5060 scope.go:122] "RemoveContainer" containerID="bd014a24da78cd6e2a035a81da50a2cb0bb331308a26e20d77f5fbf21423ece5"
Oct 30 12:47:04.306184 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.306168    5060 scope.go:122] "RemoveContainer" containerID="b997af5b8b0ce762c50d7fe5220ee9901ffe4b792379aa91865cd0a517e35301"
Oct 30 12:47:04.314417 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.314398    5060 scope.go:122] "RemoveContainer" containerID="279be4a20b5bca58c90d8b1e977db0a39d9e7f842fa19a641501702e2ccd9fa1"
Oct 30 12:47:04.322760 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.322741    5060 scope.go:122] "RemoveContainer" containerID="693cb18ce1d04e679209ac7a5b7eba2fc069baea68d8e165a49fd1bc605647db"
Oct 30 12:47:04.457698 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.457676    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f8c001c8-fd7gt_e021ad8a-13c9-4f2a-8282-3a325e84c46e/revoke-keys/0.log"
Oct 30 12:47:04.640378 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:04.640306    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fff61e17-qdn47_21b66150-d4bc-424f-81e1-c98c768068e6/revoke-keys/0.log"
Oct 30 12:47:06.528513 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:06.528414    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-h4nph_c85b9afe-dc24-4cf9-8639-a59f727c4bb2/manager/1.log"
Oct 30 12:47:06.715617 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:06.715588    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lfxxm_d621ee63-03be-454b-b1b1-9e54a635433b/manager/0.log"
Oct 30 12:47:07.164028 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:07.163989    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7h64q_e1312742-65e5-4fd2-9751-0a260930048f/manager/0.log"
Oct 30 12:47:07.584259 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:07.584227    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pzz7z_ba17fa44-3f35-497d-b2ce-b4a6415511dc/manager/0.log"
Oct 30 12:47:08.015783 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:08.015748    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-h4nph_c85b9afe-dc24-4cf9-8639-a59f727c4bb2/manager/0.log"
Oct 30 12:47:08.559170 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:08.559142    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft92xg_6a346544-3796-45ea-81ee-cfba3265d767/istio-proxy/0.log"
Oct 30 12:47:09.532161 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:09.532131    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-j2htj_9c55ce47-7db4-4f00-8caa-920ee0869c2a/istio-proxy/0.log"
Oct 30 12:47:09.708067 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:09.708038    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mnfsj_f26c00f4-7dcf-42fc-bf26-bbaddc9ea873/istio-proxy/0.log"
Oct 30 12:47:10.088953 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:10.088925    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn_33aa61d0-08cb-41ba-92ef-0c2ecd809355/istio-proxy/0.log"
Oct 30 12:47:10.687305 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:10.687273    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8f76c5df-xxgbt_5c714f27-241c-4c7f-86fe-2c3eb89d0542/router/0.log"
Oct 30 12:47:19.856263 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:47:19.856214    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:47:23.494714 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:23.494680    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:23.638961 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:23.638914    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:27.294769 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:27.294734    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:27.435511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:27.435476    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:29.384869 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:29.384834    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r4xxg_63257d8b-a976-4db1-8721-98f7d7fefed8/global-pull-secret-syncer/0.log"
Oct 30 12:47:29.551774 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:29.551746    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-488kk_8c68e591-1f33-4482-9843-d0412cec6bce/connection-checker/0.log"
Oct 30 12:47:29.662883 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:29.662809    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5b47m_e3746ecb-830d-4d86-8400-21f4d2985b22/connection-checker/0.log"
Oct 30 12:47:29.777527 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:29.777497    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-x2vn2_ea88fbd7-988d-44a5-a12e-fd5e64e8da79/connection-checker/0.log"
Oct 30 12:47:29.915993 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:29.915912    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log"
Oct 30 12:47:30.356947 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:30.356915    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-105.ec2.internal_148cbd5f3b955e9e79f6d6f208c89d56/haproxy/0.log"
Oct 30 12:47:37.278580 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:37.278540    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-h4nph_c85b9afe-dc24-4cf9-8639-a59f727c4bb2/manager/1.log"
Oct 30 12:47:37.281539 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:37.281519    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-h4nph_c85b9afe-dc24-4cf9-8639-a59f727c4bb2/manager/0.log"
Oct 30 12:47:37.439921 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:37.439895    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lfxxm_d621ee63-03be-454b-b1b1-9e54a635433b/manager/0.log"
Oct 30 12:47:37.636274 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:37.636193    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:47:37.759363 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:37.759326    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:47:37.777307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:37.777278    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7h64q_e1312742-65e5-4fd2-9751-0a260930048f/manager/0.log"
Oct 30 12:47:38.478147 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:38.478113    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:38.522764 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:38.522737    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pzz7z_ba17fa44-3f35-497d-b2ce-b4a6415511dc/manager/0.log"
Oct 30 12:47:39.044723 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:39.044694    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:39.116333 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:39.116284    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:47:39.230430 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:39.230394    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:47:40.069106 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.069074    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:40.553757 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.553724    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:40.708802 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.708766    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"]
Oct 30 12:47:40.709229 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709213    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a330a62e-dc32-48cb-8ebd-2afafbf511df" containerName="revoke-keys"
Oct 30 12:47:40.709284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709232    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330a62e-dc32-48cb-8ebd-2afafbf511df" containerName="revoke-keys"
Oct 30 12:47:40.709284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709244    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main"
Oct 30 12:47:40.709284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709249    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main"
Oct 30 12:47:40.709284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709260    5060 cpu_manager.go:414] "containerMap: removing container" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy"
Oct 30 12:47:40.709284 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709267    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892354bd-b329-4d9f-8b3e-a2eda63c55af" containerName="istio-proxy"
Oct 30 12:47:40.709514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709351    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330a62e-dc32-48cb-8ebd-2afafbf511df" containerName="revoke-keys"
Oct 30 12:47:40.709514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.709361    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32f971d-ffd4-4f01-b040-a6ff204125c5" containerName="main"
Oct 30 12:47:40.712925 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.712904    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.728627 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.728595    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c97ck\"/\"openshift-service-ca.crt\""
Oct 30 12:47:40.729721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.729697    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"]
Oct 30 12:47:40.737848 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.737823    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c97ck\"/\"kube-root-ca.crt\""
Oct 30 12:47:40.740575 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.740555    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c97ck\"/\"default-dockercfg-nn6n8\""
Oct 30 12:47:40.775997 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.775970    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:40.790511 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.790485    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-proc\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.790635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.790520    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-lib-modules\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.790635 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.790551    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-podres\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.790759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.790674    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2w65\" (UniqueName: \"kubernetes.io/projected/8c46358f-8f70-49cb-8b1d-1537454eb250-kube-api-access-c2w65\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.790759 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.790703    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-sys\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.891895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.891818    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2w65\" (UniqueName: \"kubernetes.io/projected/8c46358f-8f70-49cb-8b1d-1537454eb250-kube-api-access-c2w65\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.891895 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.891864    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-sys\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.891930    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-sys\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.891956    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-proc\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.891978    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-lib-modules\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892114 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.892013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-podres\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.892110    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-proc\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.892134    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-lib-modules\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.892320 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.892139    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c46358f-8f70-49cb-8b1d-1537454eb250-podres\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:40.916560 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:40.916524    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2w65\" (UniqueName: \"kubernetes.io/projected/8c46358f-8f70-49cb-8b1d-1537454eb250-kube-api-access-c2w65\") pod \"perf-node-gather-daemonset-wtmmp\" (UID: \"8c46358f-8f70-49cb-8b1d-1537454eb250\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:41.028655 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.028624    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:41.155779 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.155697    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-nwx29_a4ebbfa3-4272-4095-8a57-a55146be5135/cluster-monitoring-operator/0.log"
Oct 30 12:47:41.337469 ip-10-0-142-105 kubenswrapper[5060]: W1030 12:47:41.337385    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c46358f_8f70_49cb_8b1d_1537454eb250.slice/crio-f96a89a27754246d1a5e74948775ba811ba5ee6ec127800c7f399bc2a3361d96 WatchSource:0}: Error finding container f96a89a27754246d1a5e74948775ba811ba5ee6ec127800c7f399bc2a3361d96: Status 404 returned error can't find the container with id f96a89a27754246d1a5e74948775ba811ba5ee6ec127800c7f399bc2a3361d96
Oct 30 12:47:41.338345 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.338305    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"]
Oct 30 12:47:41.662514 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.662474    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp" event={"ID":"8c46358f-8f70-49cb-8b1d-1537454eb250","Type":"ContainerStarted","Data":"cbe8fdea93e66b1e2d53d83a2f8e5ed5c4185ffd78cbc37dedbc6ea80b4be3cf"}
Oct 30 12:47:41.662721 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.662618    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp" event={"ID":"8c46358f-8f70-49cb-8b1d-1537454eb250","Type":"ContainerStarted","Data":"f96a89a27754246d1a5e74948775ba811ba5ee6ec127800c7f399bc2a3361d96"}
Oct 30 12:47:41.662803 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.662762    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:41.707351 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.707304    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp" podStartSLOduration=1.707291125 podStartE2EDuration="1.707291125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:41.705339022 +0000 UTC m=+4665.500223518" watchObservedRunningTime="2026-10-30 12:47:41.707291125 +0000 UTC m=+4665.502175615"
Oct 30 12:47:41.884132 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.884044    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7d764_4e33e6c3-9095-4ac7-8df7-c97744686f2c/node-exporter/0.log"
Oct 30 12:47:41.998352 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:41.998322    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7d764_4e33e6c3-9095-4ac7-8df7-c97744686f2c/kube-rbac-proxy/0.log"
Oct 30 12:47:42.026325 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:42.026299    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:42.114936 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:42.114912    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7d764_4e33e6c3-9095-4ac7-8df7-c97744686f2c/init-textfile/0.log"
Oct 30 12:47:42.515726 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:42.515694    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:47:42.611472 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:42.611417    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:47:43.209592 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:43.209560    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:43.650710 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:43.650630    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:47:43.710734 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:43.710698    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:43.751397 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:43.751365    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:47:44.646443 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:44.646408    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:45.157412 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:45.157380    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:45.483025 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:45.482980    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:46.612298 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:46.612269    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:47.250307 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:47.250272    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:47:47.369722 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:47.369693    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:47:47.526815 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:47.526741    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-b5tmq_c173a778-6503-4ebc-9bd4-750a98b0f9d9/networking-console-plugin/0.log"
Oct 30 12:47:47.685851 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:47.685812    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-wtmmp"
Oct 30 12:47:48.010367 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:48.010330    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:48.397654 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:48.397579    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kvk9h_c3a2eebf-58c9-42a6-8615-4fde0a678b6f/manager/0.log"
Oct 30 12:47:48.498893 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:48.498854    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-64c77_6530bee4-f8d4-4326-aac2-8cd023b83c91/manager/0.log"
Oct 30 12:47:48.511610 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:48.511584    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:48.820278 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:48.820234    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/2.log"
Oct 30 12:47:48.830152 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:48.830129    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scqxd_c4c2b9dd-c717-4f1c-9b69-bc9e61e2df18/console-operator/3.log"
Oct 30 12:47:49.349079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:49.349044    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8t6zk_08df536b-fbf0-44f9-ae9f-ef64ee725386/manager/0.log"
Oct 30 12:47:49.859652 ip-10-0-142-105 kubenswrapper[5060]: E1030 12:47:49.859610    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x9lhg" containerName="extractor"
Oct 30 12:47:49.871986 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:49.871959    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j2qgw_97e52208-b29d-47e3-b683-f0e835f8e049/server/0.log"
Oct 30 12:47:50.298578 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:50.298551    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:51.408599 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:51.408562    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wlb56_436af639-6583-46e3-89d5-665f2c870571/proxy/0.log"
Oct 30 12:47:51.716889 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:51.716860    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-7mg8k_5d1d8529-d285-41d3-b179-6f3343059567/volume-data-source-validator/0.log"
Oct 30 12:47:52.558873 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:52.558840    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft92xg_6a346544-3796-45ea-81ee-cfba3265d767/istio-proxy/0.log"
Oct 30 12:47:53.444956 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:53.444928    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft92xg_6a346544-3796-45ea-81ee-cfba3265d767/istio-proxy/0.log"
Oct 30 12:47:54.886484 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:54.886418    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:54.922079 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:54.922049    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgz8g_1d000fdc-e5ed-4fce-bb1e-6715932a37b1/dns/0.log"
Oct 30 12:47:55.039897 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:55.039864    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:55.043813 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:55.043786    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgz8g_1d000fdc-e5ed-4fce-bb1e-6715932a37b1/kube-rbac-proxy/0.log"
Oct 30 12:47:55.163515 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:55.163425    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lsq8_5815b6d3-0bdd-45b2-a673-5f493146c0f8/dns-node-resolver/0.log"
Oct 30 12:47:55.188370 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:55.188345    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft92xg_6a346544-3796-45ea-81ee-cfba3265d767/istio-proxy/0.log"
Oct 30 12:47:56.995273 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:56.995237    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dhfxh_46099da2-dc06-45b7-bc3c-139b84c896e0/node-ca/0.log"
Oct 30 12:47:57.076138 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:57.076112    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-j2htj_9c55ce47-7db4-4f00-8caa-920ee0869c2a/istio-proxy/0.log"
Oct 30 12:47:57.136645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:57.136614    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mnfsj_f26c00f4-7dcf-42fc-bf26-bbaddc9ea873/istio-proxy/0.log"
Oct 30 12:47:57.261899 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:57.261820    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn_33aa61d0-08cb-41ba-92ef-0c2ecd809355/istio-proxy/0.log"
Oct 30 12:47:57.458808 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:57.458780    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8f76c5df-xxgbt_5c714f27-241c-4c7f-86fe-2c3eb89d0542/router/0.log"
Oct 30 12:47:57.605860 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:57.605774    5060 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-58qds_ec657dda-01bf-42d4-96de-1f9bb920dfff/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:58.494337 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:58.494276    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-j2htj_9c55ce47-7db4-4f00-8caa-920ee0869c2a/istio-proxy/0.log"
Oct 30 12:47:58.561645 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:58.561606    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mnfsj_f26c00f4-7dcf-42fc-bf26-bbaddc9ea873/istio-proxy/0.log"
Oct 30 12:47:58.704545 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:58.704512    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn_33aa61d0-08cb-41ba-92ef-0c2ecd809355/istio-proxy/0.log"
Oct 30 12:47:58.912087 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:58.911978    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8f76c5df-xxgbt_5c714f27-241c-4c7f-86fe-2c3eb89d0542/router/0.log"
Oct 30 12:47:59.115825 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:59.115770    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft92xg_6a346544-3796-45ea-81ee-cfba3265d767/istio-proxy/0.log"
Oct 30 12:47:59.654932 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:59.654900    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-j2htj_9c55ce47-7db4-4f00-8caa-920ee0869c2a/istio-proxy/0.log"
Oct 30 12:47:59.712210 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:59.712180    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-mnfsj_f26c00f4-7dcf-42fc-bf26-bbaddc9ea873/istio-proxy/0.log"
Oct 30 12:47:59.830388 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:47:59.830336    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-bg4vn_33aa61d0-08cb-41ba-92ef-0c2ecd809355/istio-proxy/0.log"
Oct 30 12:48:00.035885 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.035848    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"]
Oct 30 12:48:00.045971 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.045944    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"]
Oct 30 12:48:00.048403 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.048373    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55322a1-f7n7f"]
Oct 30 12:48:00.051039 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.051020    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c001c8-fd7gt"]
Oct 30 12:48:00.060598 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.060573    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8f76c5df-xxgbt_5c714f27-241c-4c7f-86fe-2c3eb89d0542/router/0.log"
Oct 30 12:48:00.337369 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.337285    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c276f3f-baf0-4bc2-b548-0d48554882c9" path="/var/lib/kubelet/pods/8c276f3f-baf0-4bc2-b548-0d48554882c9/volumes"
Oct 30 12:48:00.337958 ip-10-0-142-105 kubenswrapper[5060]: I1030 12:48:00.337938    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e021ad8a-13c9-4f2a-8282-3a325e84c46e" path="/var/lib/kubelet/pods/e021ad8a-13c9-4f2a-8282-3a325e84c46e/volumes"